Finding latency issues (stalls) in embedded Linux systems

Posted by camh on Stack Overflow See other posts from Stack Overflow or by camh
Published on 2010-04-26T06:16:31Z Indexed on 2010/04/26 6:23 UTC
Read the original article Hit count: 468

Filed under:

linux

|

embedded

|

latency

|

tools

I have an embedded Linux system running on an Atmel AT91SAM9260EK board on which I have two processes running at real-time priority. A manager process periodically "pings" a worker process using POSIX message queues to check the health of the worker process. Usually the round-trip ping takes about 1ms, but very occasionally it takes much longer - about 800ms. There are no other processes that run at a higher priority.

It appears the stall may be related to logging (syslog). If I stop logging the problem seems to go away. However it makes no difference if the log file is on JFFS2 or NFS. No other processes are writing to the "disk" - just syslog.

What tools are available to me to help me track down why these stalls are occurring? I am aware of latencytop and will be using that. Are there some other tools that may be more useful?

Some details:

Kernel version: 2.6.32.8
libc (syslog functions): uClibc 0.9.30.1
syslog: busybox 1.15.2

© Stack Overflow or respective owner

Related posts about linux

apt-get install and update fail

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I've got a problem with apt-get update and apt-get install ... commands . every time update or installing fails and errors are : Get:1 http://dl.google.com stable Release.gpg [198B] Ign http://dl.google.com/linux/chrome/deb/ stable/main Translation-en_US Get:2 http://dl… >>> More
kernel module compiling error

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
sh@ubuntu:/home/ccpp/helloworld$ make gcc-4.6 -O2 -DMODULE -D_KERNEL_ -W -Wall -Wstrict-prototypes -Wmissing-prototypes -isystem /lib/modules/`uname -r`/build/include -c -o hello-1.o hello-1.c hello-1.c:4:0: warning: "MODULE" redefined [enabled by default] <command-line>:0:0: note: this is… >>> More
Build-Essentials installation failing

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I am having trouble accessing the several critical header files that show to be a part of the build process. The "Ubuntu Software Center" shows "Build Essentials" as installed: Next I did the following two commands, which did not improve the problem: ~$ sudo apt-get install build-essential [sudo]… >>> More
Updating Debian kernel

as seen on Super User - Search for 'Super User'
I'm trying to update my Debian machine to 2.6.32-46 (which is the new stable). However, after doing apt-get update my apt-cache search linux-image shows me: linux-headers-2.6.32-5-486 - Header files for Linux 2.6.32-5-486 linux-headers-2.6.32-5-686-bigmem - Header files for Linux 2.6.32-5-686-bigmem linux-headers-2… >>> More
Serial connection over a single USB cable (Windows to linux, or linux to linux)

as seen on Server Fault - Search for 'Server Fault'
I'm helping out with a project for an embedded device that only has USB and no serial. This device is running Linux. These days, when we need to connect to a serial port on a device we typically use a USB to serial adapter (on something like a phone system or a load balancing device, etc). I would… >>> More

Related posts about embedded

New Product: Oracle Java ME Embedded 3.2 – Small, Smart, Connected

as seen on Oracle Blogs - Search for 'Oracle Blogs'
The Internet of Things (IoT) is coming. And, with todays launch of the Oracle Java ME Embedded 3.2 product, Java is going to play an even greater role in it. Java in the Internet of Things By all accounts, intelligent embedded devices are penetrating the world around us – driving industrial… >>> More
Embedded Office Application - How to know whether the app is running or the object is embedded

as seen on Stack Overflow - Search for 'Stack Overflow'
I am adding an Excel Worksheet object inside Word. Excel has a COM addin attached to it. Any idea how one can know whether Excel is running independently or as an embedded object? The COM Addin attached to Excel loads when the embedded Excel object is activated (double-clicked) inside Word. I am… >>> More
10 Reasons Why Java is the Top Embedded Platform

as seen on Oracle Blogs - Search for 'Oracle Blogs'
With the release of Oracle ME Embedded 3.2 and Oracle Java Embedded Suite, Java is now ready to fully move into the embedded developer space, what many have called the "Internet of Things". Here are 10 reasons why Java is the top embedded platform. 1. Decouples software development… >>> More
Junit with Embedded Glassfish fails - JMS Resource Adapter should be EMBEDDED

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm trying to test a session bean (NetBeans 6.8, Glassfish V3). Unfortunately, the embedded glassfish is unable to start properly, as it tries to connect to a remote JMS Provider (at localhost:7676): $ ant test ... [junit] Mar 23, 2010 12:13:51 PM com.sun.messaging.jms.ra.ResourceAdapter start [junit]… >>> More
pass value from embedded function into conditional of page the embedded function is included on

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a page that includes/embeds a file that contains a number of functions. One of the functions has a variable I want to pass back onto the page that the file is embedded on. <?php include('functions.php'); userInGroup(); if($user_in_group) { print 'user is in group'; } else { print 'user… >>> More