## 2015년 6월 26일 금요일

### Problem when using GNU Screen 'stuff' command with 40+ open tabs - runaway input to stdin

Once a month I conduct system diagnostics on about 40 machines at an IDC for a Korean food company. Because the commands I need to run (i.e. dmesg | grep -iE "criti|fail|warn|bug|") are the same for all  machines, I use a script to launch GNU Screen with dozens of tabs each ssh'ing into a different server and once connected to all machines, I use

C-a :at "#" stuff "cmd^M"

to send commands to all tabs at once. Without this feature, there is no way I could complete a system checkup on 40+ machines in less than a few hours.

I recently had to install the leap second patch for tzdata packages on 40+ RHEL6.X machines, and wanted to scp them from my local machine to all the servers in one fell swoop. Therefore I entered the following command into GNU Screen:

:at "#" stuff "scp -r archjun@10.200.250.156:/MULTIMEDIA/iso/pkg-update/tzdata^M" .

As this was a new ssh connection, I was prompted whether I wanted to continue connecting:

The authenticity of host '10.200.250.156 (10.200.250.156)' can't be established.
RSA key fingerprint is
...
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.200.250.156' (RSA) to the list of known hosts.

Obviously I wasn't about to type 'yes' into each of 40+ tabs. Therefore I stuffed yes into all 40+ GNU Screen tabs using

:at "#" stuff "yes^M"

All was fine for about 20 tabs, but the remaining tabs entered an input loop stuffing an infinite number of y's to stdin

y
y
y
y
y
y
y
...

In my .screenrc config file I have automatic logging turned on with the two lines

logfile $HOME/Documents/term_sessions/%Y-%m-%d_%0c-%n.log deflog on so that a logfile is automatically generated as soon as a new GNU Screen tab is created (hotkey C-a c). I noticed that logfiles were suddenly ballooning to hundreds of megabytes in size. I manually had to enter C-c into each of these tabs to stop the runaway input. I have never experienced anything like this when stuffing commands to all windows in GNU Screen. Has anyone else encountered similar behavior? ## 2015년 6월 14일 일요일 ### Setting up mcelog to work with systemd If you regularly observe system logs such as /var/log/messages, dmesg, or journalctl (systemd) you will eventually encounter a Machine Check Event (mce) which warns you that some kind of hardware error has occurred. For example, a common mce is caused when an incorrect bit is flipped in RAM. For server ECC memory, this is less of a problem because such bit errors can be fixed automatically. When encountering servers in the field with uptime greater than 365 days, it is not hard to find mce errors logged here and there. In the case of RHEL 5/6 machines I encounter in the field, mce errors are logged in the file /var/log/mcelog and the mcelog service runs by default. Recently I noticed that every few days the kernel ring buffer dmesg on my work laptop gives the following error: [Jun11 23:49] mce: [Hardware Error]: Machine check events logged However, when I navigate to /var/log/ I cannot see any file named mcelog. Some old posts floating around the Internet recommend redirecting mcelog to some output file, i.e. /usr/sbin/mcelog > mcelog.out but this didn't work for me. Make sure you have the mcelog package (as it is called in Arch) installed . To enable the daemon in systemd, systemctl enable mcelog. When running mcelog on a Linux machine running systemd instead of the old syslog, you need to make some changes to /etc/mcelog/mcelog.conf What led me astray was the Archwiki page on MCE Handling, which recommends uncommenting the line syslog = yes If you are running systemd you do NOT want the above setting! The problem is that systemd handles system logging through journalctl. You can follow the other suggestions in the Archwiki to run mcelog as a daemon (daemon = yes), but make sure the syslog lines are commented out. Also you need to specify an output log file for mce errors by uncommenting the following in /etc/mcelog/mcelog.conf: logfile = /var/log/mcelog Also uncomment the following in /etc/mcelog/mcelog.conf run-credentials-user = root Restart the mcelog service systemctl restart mcelog Next time a Machine Check Event occurs, it will be written to /var/log/mcelog. Here is some sample output: Hardware event. This is not a software error. MCE 0 CPU 0 BANK 5 MISC b8a0000086 ADDR ffb07500 TIME 1434034181 Thu Jun 11 23:49:41 2015 MCG status: MCi status: Error overflow Uncorrected error MCi_MISC register valid MCi_ADDR register valid Processor context corrupt MCA: corrected filtering (some unreported errors in same region) Generic CACHE Level-2 Generic Error STATUS ee0000000040110a MCGSTATUS 0 MCGCAP c07 APICID 0 SOCKETID 0 CPUID Vendor Intel Family 6 Model 69 Hardware event. This is not a software error. MCE 0 CPU 0 BANK 5 MISC 78a0000086 ADDR ffb07500 This seems to be indicating a memory error in the CPU cache. Here is my /etc/mcelog/mcelog.conf file: ## 2015년 6월 7일 일요일 ### [SOLVED] Upgrade from Python 2.7.9 to 2.7.10 breaks Brainworkshop Note: I originally made this post in June 2015, but in Dec. 2015 I found a workaround to get Brainworkshop 4.8.4/4.8.7 to work with Python 2.7.10+. Please refer to the new post at the following link: http://eatpeppershothot.blogspot.kr/2015/12/enabling-brainworkshop-487-to-work-with.html ================================================================= Recently Archlinux made python2-2.7.10-1 available in the main Arch repositories. Unfortunately, however, the Brainworkshop N-back training program doesn't work with the latest version of Python 2. In Gentoo and Arch, the brainworkshop.pyw source file is launched by the startup script /usr/bin/brainworkshop which looks like the following (I have edited it to correspond to my BW data and config files on Dropbox): After upgrading from Python 2.7.9 to 2.7.10, however, the following error message is generated: /usr/bin/brainworkshop: line 8: 5212 Segmentation fault (core dumped) python2 /usr/share/brainworkshop/brainworkshop.pyw --configfile$BRW_CONFIGFILE --statsfile $BRW_STATFILE --datadir$BRW_DATADIR

The actual Python 2 source file is located at /usr/share/brainworkshop/brainworkshop.pyw (.pyw instead of .py so that a separate console window won't be launched). Even launching brainworkshop.pyw directly without any option flags returns Segmentation fault (core dumped). In Archlinux, core dump files are located in /var/lib/systemd/coredump, but rather than trying to analyze a dump of system memory at the time brainworkshop.pyw crashed, I decided to run gdb to see if I could get more informative error messages through a stack trace. Although gdb is normally run on C/C++ compiled executables, gdb (version 7+) will also work with Python if you invoke it as follows:

[archjun@lenovoS310 ~]$gdb /usr/bin/python2 GNU gdb (GDB) 7.9.1 Copyright (C) 2015 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/bin/python2...(no debugging symbols found)...done. Now at the gdb prompt, type run followed by the name of the python program: (gdb) run /usr/share/brainworkshop/brainworkshop.pyw Starting program: /usr/bin/python2 /usr/share/brainworkshop/brainworkshop.pyw [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Traceback (most recent call last): File "/usr/share/brainworkshop/brainworkshop.pyw", line 1023, in window = MyWindow(cfg.WINDOW_WIDTH, cfg.WINDOW_HEIGHT, caption=''.join(caption), style=style, vsync=VSYNC) File "/usr/lib/python2.7/site-packages/pyglet/window/xlib/__init__.py", line 163, in __init__ super(XlibWindow, self).__init__(*args, **kwargs) File "/usr/lib/python2.7/site-packages/pyglet/window/__init__.py", line 494, in __init__ display = get_platform().get_default_display() File "/usr/lib/python2.7/site-packages/pyglet/window/__init__.py", line 1766, in get_default_display return pyglet.canvas.get_display() File "/usr/lib/python2.7/site-packages/pyglet/canvas/__init__.py", line 82, in get_display return Display() File "/usr/lib/python2.7/site-packages/pyglet/canvas/xlib.py", line 83, in __init__ raise NoSuchDisplayException('Cannot connect to "%s"' % name) pyglet.canvas.xlib.NoSuchDisplayException: Cannot connect to "None" [Inferior 1 (process 26382) exited with code 01] So it seems like Python's pyglet module is the culprit once again in breaking Brainworkshop. In a previous post from February 2015 (pyglet 1.2 breaks Brainworkshop), I described a way to get Brainworkshop working again by making a few edits to the /usr/share/brainworkshop/brainworkshop.pyw source file. I have played around with changing some of the parameters in line 1024 window = MyWindow(cfg.WINDOW_WIDTH, cfg.WINDOW_HEIGHT, caption=''.join(caption), style=style, vsync=VSYNC) but I have yet to succeed in getting Brainworkshop to work with Python 2.7.10. Workaround Luckily, BW still works with Python 2.7.9, so I downgraded to 2.7.9 with pacman -U /var/cache/pacman/pkg/python2-2.7.9-1-x86_64.pkg.tar.xz (if you don't have this package in your pacman package cache, you can still download it from the Arch Linux Archive) and added python2 to IgnorePkg in /etc/pacman.conf as a temporary workaround. Update 2015-09-07 python2-pyglet in Archlinux was recently upgraded from 1.2.3-1 to 1.2.4-1. Now when trying to run Brainworkshop with Python 2.7.10 I get the following error in brainworkshop.pyw when debugging with gdb: [archjun@latitude630 ~]$ gdb /usr/bin/python2
GNU gdb (GDB) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/python2...(no debugging symbols found)...done.
(gdb) run /usr/share/brainworkshop/brainworkshop.pyw
Starting program: /usr/bin/python2 /usr/share/brainworkshop/brainworkshop.pyw

Program received signal SIGSEGV, Segmentation fault.
0x00007fffde270664 in gdk_pixbuf_loader_write () from /usr/lib/libgdk_pixbuf-2.0.so.0

It looks like we are no longer getting errors from pyglet like we did above with pyglet 1.2.3, but now we are having problems with libgdk_pixbuf-2.0.so.0. Let's take a look at this file:

[archjun@latitude630 ~]$ls -al /usr/lib/libgdk_pixbuf-2.0.so.0 lrwxrwxrwx 1 root root 29 Aug 20 20:32 /usr/lib/libgdk_pixbuf-2.0.so.0 -> libgdk_pixbuf-2.0.so.0.3100.6 Apparently libgdk_pixbuf-2.0.so.0 is a symlink from the file /usr/lib/libgdk_pixbuf-2.0.so.0.3100.6 What package owns libgdk_pixbuf-2.0.so.0.3100.6? [archjun@latitude630 lib]$ sudo pacman -Qo libgdk_pixbuf-2.0.so.0.3100.6
/usr/lib/libgdk_pixbuf-2.0.so.0.3100.6 is owned by gdk-pixbuf2 2.31.6-1

[archjun@latitude630 lib]\$ sudo pacman -Ss gdk-pixbuf2
extra/gdk-pixbuf2 2.31.6-1 [installed]

According to gdb, the function that is causing trouble is called gdk_pixbuf_loader_write (). Looking at the Gnome GDK-PixBuf Reference Manual, we can find more detailed information on the C function gdk_pixbuf_loader_write (), but I'm not familiar with C so I am not sure where to start debugging this issue.

A google search for gdk-pixbuf2 gdk_pixbuf_loader_write () segfault returns an interesting (but dated) google code issues page on github that discusses problems with pyglet decoding images using gdk-pixbuf2. One proposed workaround is to try PIL, pyllow or pypng for image decoding instead of gdk-pixbuf2.

Opening /usr/share/brainworkshop/brainworkshop.pyw with a text editor such as vim or emacs quickly reveals that there are no explicit calls to gdk functions in the Python code. We know that pyglet can use gdk-pixbuf2 when it tries to decode images, so I searched for invocations of pyglet.image and found 6 results at lines 1031, 2304, 2327, 4622, 4626, and 4628, respectively:

if sys.platform == 'linux2':
...

...