Tkinter and Unicode emoji variation

For a small chess program I created a small dictionary I use to convert an internal direction to an arrow glyph for display:

# large arrow glyphs (Unicode)

ARROWGLYPH = {
	 "n": "\u2B06",		# straight up
	"ne": "\u2B08",		# up and right
	 "e": "\u2B95",		# straight right
	"se": "\u2B0A",		# down and right
	 "s": "\u2B07",		# straight down
	"sw": "\u2B0B",		# down and left
	 "w": "\u2B05",		# straight left
	"nw": "\u2B09",		# up and left
	"k1": "\u2B08",		# up two, right one
	"k2": "\u2B08",		# up one, right two
	"k3": "\u2B0A",		# down one, right two
	"k4": "\u2B0A",		# down two, right one
	"k5": "\u2B0B",		# down two, left one
	"k6": "\u2B0B",		# down one, left two
	"k7": "\u2B09",		# up one, left two
	"k8": "\u2B09",		# up two, left one
	"P2": "\u2B06",		# up two
	"p2": "\u2B07",		# down two
	"ck": "\u2B95",		# right two
	"cq": "\u2B05",		# left two
	}

When it comes time to display any of these, I change the text of a tkinter button to one of these Unicode characters. My problem is that these do not always display the way I expect them to. Specifically, the Unicode characters associated with “w”, “s” and “n” appear as narrow arrows within boxes, rather than as the wide arrows I want.

It seems that the Unicode values of “\u2b05”, “\u2b06” and “\u2b07” are emoji characters that can have variants. This could be what is happening, although my understanding is that a variant requires a following variant character (V15 or V16) to trigger the display of an alternate glyph. I can’t see anywhere in my code any attachment of a variant character to any arrow character. Nor does any examination of a value before or after attaching it to a button show anything other than the expected single Unicode character.

I do not believe that it would help to switch to narrow arrows altogether, as from what I can tell the three affected characters are also emoji.

Curiously to me, there does seem to be a way around this by altering the way I initialize the buttons. My goal is to create a button inside a label frame. There is only one button in each of 64 chessboard frames. My idea here is to dynamically alter the title of the frame to show information about the frame, such as its name and the number of black or white pieces attacking it. Details are probably not important, since all that seems to work just fine.

If I initialize each button’s text to a null string, the problem of differing arrow glyphs appears:

def makeSquare(parent, file, rank):
	'''make one chessbaord square'''
	# the name and color of this square
	sqname = CM.fr2sq( file, rank )
	sqcolor = CLR_LITESQ if (file%2 + rank%2)%2 else CLR_DARKSQ

	# we'll make the containing frame with the label non-null
	# - this will make the label as large as it will ever be
	frame = tk.LabelFrame( parent,
		bg=sqcolor,
		text=sqname,
		labelanchor=tk.N,
		borderwidth=0,
		font=(FNT_TEXT),
	)
	# this arrangement puts white at bottom of chessboard
	frame.grid( column=file-1, row=8-rank )
	# we need them later
	frame.dfltcolor = sqcolor
	frame.color = sqcolor
	frame.name = sqname
	# bind these events to this frame
	frame.bind("<Enter>", lambda event: enterSquare(event.widget))
	frame.bind("<Leave>",  lambda event: leaveSquare(event.widget))

	# a button inside each frame
	button = tk.Button( frame,
		text=""		,		# default text
		anchor=tk.CENTER,
		bg=sqcolor,			# it'd be nice if we could make this transparent and just color frame background
		font=( FNT_TEXT, ICO_POINTS ),
		height=1,			# this height and width makes frame look square-ish
		width=4,
		borderwidth=0,		# hide borders between buttons
	)
	# put button in frame
	button.grid(column=0, row=0, sticky="ew")
	# bind this event
	button.bind("<Button>", lambda event: hitSquare(event.widget))

	return ( sqname, frame )

Apologies for the length, but I’m trying to illustrate just how minor a change solves the display of arrow characters with varying glyphs while also causing another curiousity:

directions = ["n", "ne", "e", "se", "s", "sw", "w", "nw", "k1", "k2", "k3", "k4", "k5", "k6", "k7", "k8", "P2", "p2", "ck", "cq"]  

def makeSquare(parent, file, rank):
	'''make one chessbaord square'''
	# the name and color of this square
	sqname = CM.fr2sq( file, rank )
	sqcolor = CLR_LITESQ if (file%2 + rank%2)%2 else CLR_DARKSQ

	# we'll make the containing frame with the label non-null
	# - this will make the label as large as it will ever be
	frame = tk.LabelFrame( parent,
		bg=sqcolor,
		text=sqname,
		labelanchor=tk.N,
		borderwidth=0,
		font=(FNT_TEXT),
	)
	# this arrangement puts white at bottom of chessboard
	frame.grid( column=file-1, row=8-rank )
	# we need them later
	frame.dfltcolor = sqcolor
	frame.color = sqcolor
	frame.name = sqname
	# bind these events to this frame
	frame.bind("<Enter>", lambda event: enterSquare(event.widget))
	frame.bind("<Leave>",  lambda event: leaveSquare(event.widget))

	ndx = directions[ (file+rank) % len(directions) ]
	textval = ARROWGLYPH[ ndx ]

	# a button inside each frame
	button = tk.Button( frame,
		text=textval,		# default text
		anchor=tk.CENTER,
		bg=sqcolor,			# it'd be nice if we could make this transparent and just color frame background
		font=( FNT_TEXT, ICO_POINTS ),
		height=1,			# this height and width makes frame look square-ish
		width=4,
		borderwidth=0,		# hide borders between buttons
	)
	# put button in frame
	button.grid(column=0, row=0, sticky="ew")
	# bind this event
	button.bind("<Button>", lambda event: hitSquare(event.widget))
	# now blank the button's text
	button.config( text="" )

	return ( sqname, frame )

I first did this as a debug step, with the idea of starting out by displaying an arrow on each square. If the “wrong” arrows appeared for some directions, I imagined that would spark another idea of some kind. I was a bit surprised when the arrows looked just fine. Subsequent uses of them during later execution also looked fine. I saw no unexpected glyphs.

I don’t know (yet) how much of that is necessary, but somehow initializing each button with a Unicode arrow character prevents the emoji arrows from displaying incorrectly later. It may be that any non-null text at all would work, but I haven’t experimented that far yet.

There is still something curious going on. When doing this, the glyphs for the chess pieces themselves appear differently. The are still recognizably what they should be, but they are lighter. Their outlines are not as thick, and they are not quite so detailed. It’s as if they too are now showing a variant form - the chess piece glyphs are also in a Unicode variant block - even though I didn’t ask for one.

Is there an explanation somewhere of what’s going on with these Unicode characters? Am I misunderstanding something obvious? Doing something wrong?

I’m far from an expert in this area. The only thing I can recommend is trying out other fonts to see if it’s a font issue?

An approach worth trying. I had set the original font I was using to “TkFixedFont”, but changing the font to “Arial” had no effect. All the glyphs were similar, but the same “arrow in square” problem appeared. I didn’t try any other fonts, as it seems that whatever the root cause is, it’s consistent across fonts.

I have now tried initializing each button with arbitrary non-null text of one or more characters. That has no effect in solving the problem. Reverting to initializing each button with an arrow character and then changing it to an empty string before actually showing it eliminates the unexpected arrow glyph problem and introduces the unexpected chess piece glyphs.

Though I now do this initialization in a somewhat neater way than the debugging approach I used above, the result is the same. So something about those arrow characters seems to be necessary.

I finally noticed that there is a way to upload images. This is a small (5K) slice of a screenshot ot the mis-behaving arrows.

Another small screenshot, this one showing the correct arrows (or at any rate the ones I want to see), but with a slightly altered chess glyph.

I decided to try initializing each button to the same direction rather than one from an array of them. This image shows what happens when “n”, “s”, “P2” or “p2” (ie, verticals, “\u2B06” or “\u2B07”) is used as that direction. Almost correct, but still off to the west.

Initializing all the buttons to any of the directions “ne”, “se”, “sw”, “nw”, all the “k1-8” (ie., diagonals) result in the same output as the second image above. Correct arrows in all directions with an unexpected shift in the piece glyphs. The same happens when I use “e” or “ck”.

When “w” or “cq” is used to initialze each button, the result is the same as in the first image, wrong in three directions..Something else unexpected happens, shown next:

For unknown (but consistent) reasons, the glyph for the black pawn changes. It resembles the Unicode glyph for a so-called neutral pawn (“\u1FA05”), but with the black/white sides reversed.

For the time being I may be able to cover up this problem of unexpected glyphs occurring by initializing each button’s text to the Unicode character for a diagonal direction and then changing that to a null string before displaying it. I’d prefer a real fix that began with understanding why any of these glyphs are displayed the way they are.

What OS are you running, as that often affects font behaviors?

This hardware is an HP desktop machine (i7-6700T CPU) with a QHD monitor. The software is Python 3.11 and Windows 10 Home (64-bit).

I have found that it’s sufficient to initialize just one button to a diagonal arrow to cause all of them to behave “properly”. It doesn’t hurt to initialize all of them, and in that case I don’t need to check which button is being initialized. But it doesn’t seem necessary.

I am surprised that you have this problem on Windows. I printed ARROWGLYPH.values() in IDLE Shell and see all solid arrows, [‘:up_arrow:’, ‘⬈’, ‘⮕’, ‘⬊’, ‘:down_arrow:’, ‘⬋’, ‘:left_arrow:’, ‘⬉’, ‘⬈’, ‘⬈’, ‘⬊’, ‘⬊’, ‘⬋’, ‘⬋’, ‘⬉’, ‘⬉’, ‘:up_arrow:’, ‘:down_arrow:’, ‘⮕’, ‘:left_arrow:’], like I do here in FireFox. I am using Source Code Pro. I open the IDLE Settings dialog and selected and applied several different fonts (25?). Nearly all had similar unboxed arrows, with variations in the stem width, sometimes within a font. Some bold or semi-bold fonts did not display a few of them, replacing them with the generic box with question mark. Those few looked to be the ones you had a problem with. Then … Segoe UI Emoji showed the same codepoints with thin arrows in boxes as in your images. That is my only emoji font and the only one with boxed arrows.

I’m sorry; I do not know the tool you are using. As near as I can tell, you are saying that most of the fonts you tried display the arrows correctly, but the font “Segoe UI Emoji” does not.

The only two fonts I have tried are “TkFixedFont” and “Arial”, both of which showed the same artifacts. I do not know which installed font “TkFixedFont” actually maps to, but it could be an emoji font. But I will try an explicit monospace font like “Courier” and see what happens (I want monospace because the square title labels will be consistent).

For what it’s worth, I have looked at your last message with both the Edge and Firefox browsers. In them your listing of solid arrows (ie., [..]) does indeed show all the diagonal arrows and also the rightward pointing arrow without decoration. However, in both browsers the upward, downward, and leftward pointing arrows are rendered as white arrows within blue boxes. I believe this is a variant form of these emoji arrows.

It is not clear to me at this point whether all the arrow characters have a variant character attached but rendering ignores it if there is no defined variant, or if just these three are somehow treated differently. I will try to experiment to see which is the case, although I’m not sure how to do that just yet. Der Google may be my friend here.

Okay, the fonts “Courier”, “Consolas”, and “Lucida” all show the same mis-behaving arrows and black pawn glyph when I use “w” as the single initializing arrow. When I print out various characteristics of characters using this:

def showUni(move, ch):
	print( f"\n{move=}" )
	print( f"Character: {ch}" )
	print( f"     Name: {unicodedata.name(ch)}" )
	print( f" Category: {unicodedata.category(ch)}" )
	print( f"Combining: {unicodedata.combining(ch)}" )
	print( ch.encode('UTF-8') )

both before (buttton.config(text=textval)) and after (button.cget(“text”)) assigning a Unicode character to a button’s ‘text’ attribute, the result is the same both times.

Still, it seems as if a variant specifier (VS15 or VS16) is getting attached somewhere along the line.

IDLE is the IDE that comes with Python if you have tkinter installed. py -m idlelib should start it.

All the fonts you listed behave correctly for me when displayed in a Text widget. Perhaps tk handles fonts differently in buttons. I don’t know anything more.

Thank you. I will have to investigate IDLE more thoroughly. For the moment all I have done is enter lines like print( “\u2b05” ). Yes, all eight arrow characters appear properly when I do that.

I also tried to look at the source code of this page to see if any variant markers appeared in the text of your message. Whether they did or not might have supplied a clue as to where they were coming from. But while looking a the source did show a lot of boilerplate code regarding how the page looked, when it came to the actual content this text appears instead:

“HTML content omitted because you are logged in or using a modern mobile device.”

So that seems a blind alley. Perhaps I will have better luck with IDLE.

In the meantime there is this:

https://www.unicode.org/emoji/charts/emoji-variants.html

which shows that the three characters “leftwards black arrow”, “upwards black arrow” and “downwards black arrow” and illustrates them as undecorated arrows if followed by a V15 “monochrome text” variant code, (eg.," \u2b05\ufe0e") or a white arrow in blue square if followed by a V16 “emoji color” variant code (eg., “\u2b05\ufe0f”). It’s not completely clear from this document what should happen if there is no variant code specified - which gives me an idea to try next.

Huh, one might think that appending a VS15 (“\uFE0D”) character to every arrow glyph character would force it to be rendered as monochrome text. All it really seems to do is cause an invisible but still space-taking text character to be rendered as well. So the arrows are no longer in the center of each square but instead pushed to the left.

One approach that does seem to work is to use characters that have no variant glyphs. The various Wingding code blocks have several sets of such arrow codepoints that do not have variant glyphs. One such that is rendered the way I first hoped is:

ARROWGLYPH = {
	 "n": "\U0001F879",		# straight up
	"ne": "\U0001F87D",		# up and right
	 "e": "\U0001F87A",		# straight right
	"se": "\U0001F87E",		# down and right
	 "s": "\U0001F87B",		# straight down
	"sw": "\U0001F87F",		# down and left
	 "w": "\U0001F878",		# straight left
	"nw": "\U0001F87C",		# up and left
	"k1": "\U0001F87D",		# up two, right one
	"k2": "\U0001F87D",		# up one, right two
	"k3": "\U0001F87E",		# down one, right two
	"k4": "\U0001F87E",		# down two, right one
	"k5": "\U0001F87F",		# down two, left one
	"k6": "\U0001F87F",		# down one, left two
	"k7": "\U0001F87C",		# up one, left two
	"k8": "\U0001F87C",		# up two, left one
	"P2": "\U0001F879",		# up two
	"p2": "\U0001F87B",		# down two
	"ck": "\U0001F87A",		# right two
	"cq": "\U0001F878",		# left two
	}

A bonus of sorts is that the chess piece symbols are rendered without any shift in appearance. So far I haven’t found any official document that even hints that they might have variants, but pretty clearly they do.