After writing this entry, I was thinking how we can improve 'bt' command of the crash utility.
I wrote an LDAS (Linux Dump Analysis Script) of Alicia, btx.ldas, as an alternative of 'bt'. (btx.ldas is available at this URL as usual.)
Roughly speaking, btx.ldas is a filter of 'bt' command results. The basic idea is that there are return addresses on the right most colums and that means there should be call instructions just before the address. Furthermore, it's possible to check whether a supposed return address is really correct can by seeing if the operand of a call instruction is the address of the next stack frame or not. That's what btx.ldas does basically. Of couse, there are several technical issues, but as shown in the following, this approach looks to work. :)
The below examples are results by 'btx.ldas' and crash 'bt' command of the same "sendmail" process taken on my PentiumD 930 / FedoraCore 6 (x86_64) box.
(1) crash 'bt' command example
Among the 22 stack frames shown below, there are only 6 correct ones as I wrote in this entry. btx.ldas checks five bytes (size of a call instruction) before the rightmost addresses colored in blue in each stack frame.
(2) btx.ldas result
Below is an example using btx.ldas and Alicia-1.1.5. Extra 16 frames above were suppressed.
Recently I read an interesting article about a difference of business structures between western countries and Japan. The article was an interview of a CRM software company selling president.
The essential point of the interview for me was like the following:
In order to exploit CRM software enough, a company must optimize its business activities company wide not per business division basis. For example, a design center of its products and factories must cooperate shooting for the maximum benefit of the company not each business division. Quite a lot of Japanese companies have a problem from that point of view. Simply speaking, it's a sectionalism among business divisions in one company.
The president talked about the strategy and foresight of his company more, but for me the above was interesting. Actually, that story reminded me of what an executive of my company said at a kickoff meeting of this fiscal year about a month ago. I'm working for a company with a quite traditional business structure, and a lot of business divisions inside the company are always competing with with each other for their own budgets and benefits. It's like a bureau of something. :(
BUT!
The executive finally began to say, "Think company wide optimization, not per business unit basis." Furthermore, about two weeks later of the kickoff meeting, I heard a news that the executive would be the president of my company soon. :o
The above story also reminded me of a similar issue I wrote before. I think a big company can be regarded as a society or a community which consists of multiple organizations with different stakes. Here, the organizations are like species with different *genes* carriers. If the genes are essentially different and competing with each other, the theory of evolution says that it's not stable that those species cooperate each other. Thus, the problem is that if employees of my company are sharing the same gene or not. I joined my company about three years ago after 10 years of life at other company, and for me their mentality looks like almost uniform one. So, I believe the competing divisions share the same gene. :)
BTW, the executive of my company is Mr. Yamashita. Hey, Yamashita-san, I'm really looking forward to seeing your strong leadership. :)
As we had a yakiniku party yesterday, we celebrated the anniversary by a small cake and tea party. :)
We bought them at a cake shop "L'ATELIER DE T" in Toyohashi city. http://home-eat.seesaa.net/article/33427197.html http://www.planets.co.jp/select/file16.phtml
The owner chef seems to have worked as the chief pâtissier of the Hotel Okura, Hamamatsu. My choice was the lemon tarte above, and I like it. My mother in law also loved her Mont Blanc. But, surprisingly enough for me, my wife and my father in law didn't like their cakes.
BTW, when we stayed at my home town, my mother also celebrated us by a small cake party. My choice was a blue berry tarte below. :)
Today's dinner was a Yakiniku party (barbecue party) . :) It's almost 3 years since I went to a yakiniku restaurant.
The restaurant seems to be very famous and they indeed offer very good meats. :) Here are several articles about the restaurant, Yakiniku Restaurant Hanada.
BTW, about 4 years ago, I went to this yakiniku restaurant almost once a week. :o I really loved the restaurant, but now I live very far from the place.
BTW, one of my former colleague is planning to have a party at the restaurant. Hey, Saeki-kun, it's YOU! Are you reading? I'm looking forward to seeing you guys and eating together there. :)
Shimada city is located in the middle of Shizuoka prefecture, Japan. It faces to the Pacific Ocean, so they have very calm climate and it's a very good place for growing flowers like roses.
"島田市ばらの丘公園" (Shimada-si Bara no Oka Kouen) is a kind of flower park dedicated for roses. They don't have an english organization name, but "Shimada City Rose Hill Garden" would be appropriate, I think.
My mother loves the garden, and she often goes to the place when she got guests. This time, it's May 2 we went there. Unfortunatelly, it's a bit earlier than the peak of this season. But, there were plenty of beautiful roses in the greenhouse. :)
Moto-chan no ItaiHanashi (もとちゃんの痛い話) ISBN 4-04-160010-3 MotokoArai (新井素子)
Well, there is no English translation of this book. In English literal translation, the title would be something like "Moto-chan's Sore Story." MotokoArai is a famous Sci-Fi/Fantasy writer in Japan, and this is her essay when she got a painful disease. As usual her writing style, it's not a mournful hospital diary but a good light essay, so we can share her various interesting thoughts.
Actually, when I went back to my home town in this Golden Week vacation, I found my mother was working on the book, and I was also captured. :o
Yokohama-ya (横濱屋) is a Ramen (Chinese Noodle) restaurant chain. They have about 15 restaurants in mainly Tokyo and Kanagawa. In this sense, Yokohama-ya is different from small but good stand-alone Ramen restaurants along Koganei Kaido avenue.
My wife and I had already finished evacuating our fridge today before our trip to our home towns from tomorrow, thus tonight dinner was Ramen, again. :o
Here is my choice. Chives and Beansprouts Ramen with sliced pork. I felt it's about 1.5 starts. :(
When I read the previous issue in detail last month, I felt it's too elementary because example programs are:
Blinking an LED on the board
Sending a "Hello, World!" message infinitely
But, there are practical application examples such as a line-tracing car controlled by the V850 board in this month issue, and finally that makes sense for me. Maybe the editors were too busy for preparing the giveaway. :o
BTW, one of the article was about a u-ITRON (TOPPERS) port for the board. I'm wondering if I should go ahead for another program monitor... Hey, Furukawa-san, are you reading? What do you think about? :)
In Japan, a big vacation season is beginning from tomorrow, and my wife had a medical check before the vacation.
Now, thatsdone 3.0 is 14 weeks and 16cm tall without any troubles. :)
Today, the baby seemed to be sleeping and kept still at first, but after a few minutes of the check, woke up and was waving its hands and legs. Well, I'm not sure if the baby was saying, "Don't wake me up!" :o)
In Japan, usually March and September are farewell party seasons. So, in the beginning of this April, there was a organization restructuring at work and I had several farewell and welcome parties as usual. But, last week I got a bit shocking news that a colleague of mine would leave our organization by the end of this April. I'm not sure why it's in the end of April not March. Could be because of his new organization's convenience. Anyway, I hope he and his family live in happy harmony there, his wife's home town and also close to his home town. But, what I'm wondering is how the remaining people can fill the role which he did for these two years.
Well, the party was held at Inataya (稲田屋), Shinagawa. http://www.inataya.co.jp/store/sinagawa/index.html
It's an average class Japanese style pub, and I drank a bit too much Beer and Japanese Sake there. So, I fell over after leaving the pub, and what is worse, I hit my legs hard with something. My legs are still aching. :( Two my bosses witnessed that. :o
BTW, I didn't know it's "Inata-ya" not "Inada-ya." :o
I have to confess that I didn't know Linux kernel (general?) linkage convention well till the last Sunday. :(
I've been feeling strange we often get unknown function call entries in kernel stack traces like the below example when we are running Linux on x86 or x86_64 architecture. Among the stack trace below, entries in red cannot be found even if you looked into the source code in detail.
Finally, I understood the reason.
Below code chunk is excerpted from 'arch/x86_64/kernel/traps.c' of RHEL5 Update 0 kernel (based on linux-2.6.18). The function 'dump_trace' is the one which prints out stack traces, and note that lines from 269 to 272 say something strange. That means 'dump_trace' cannot determine boundaries of stack frames strictly. Thus, taking a stack area as if it's an array of pointers, the function prints an entry into the stack trace if the value looks like an in-kernel function. :( The reason comes from the linkage convention of Linux (x86 and x86_64, at least).
In those architectures, callee functions do NOT save any registers except ones which the callees destroy. Even the stack pointer (%sp or %rsp register). Only return addresses are saved automatically by 'call' instructions.
That's why, different from other architectures/operating systems like SPARC/Solaris, it's difficult to track back the caller functions correctly among the stack area in case of Linux/x86(x86_64). :(
Of couse, this must come from performance consideration. But still, it makes trouble shootings difficult, I think. :(
'allie' uses kprobe which is a built-in feature in RHEL4 or later. If you want to know call trees of a particular in-kernel function such as 'ip6_xmit()' but don't want to recompile the entire kernel, 'allie' will help you very much. It's available here under GPL2 as usual. :)
When you want to know function call graphs of a particular in-kernel function, for example, 'mpage_bio_submit', then you can see them by installing 'allie' with insmod parameters like the following:
Lines in red below are kprobe and allie origin trace entries.
Under Linux(x86 and x86_64) environment, we often see actually unrelated function call entries because of Linux linkage convention as I wrote here. Don't be confused.
count=3 means to show only 3 stack traces because in-kernel functions are usually called very frequently and your /var/log/messages could be full.
kprobe uses 'int3', thus we cannot catch function calls under interrupt disabled state.
Guts Ramen is another Chinese noodle restaurant close to my house. It's closer to my house than Menya Kotaro, and actually I and my wife go to Guts most frequently among Chinese noodle restaurants along Koganei-Kaido avenue. :)
They offer very good pork bone based soup Chinese noodles and very soft sliced roast porks, and both my wife and I like them. :) In addition, Guts is unique for its advertising display and chairs. Especially, chairs look like piggs and kids must like them. :)
Here is tonight dinner. :o
Wraplings, and Chinese noodle with sliced roast pork and sliced green onion.
Well, it's about the V850 CPU board I wrote before. As I'm not good at soldering, I asked one of my colleagues to assemble separated parts including a USB connector, several external pins and a crystal module. Today, finally the guy gave me completed board, and it's the Power ON ritual(?) day for the V850 board. It worked without any trouble, and you can see its LED on the upper left corner is lighting. Thanks, Furukawa-san! :)
Now, I'm considering what I should do next, but I guess I'd write a small monitor program.
BTW, I found an excellent report page about the board. (Japanese only, at this moment) I'm a pure software guy, and so I cannot compete with him. :o
Interface Magazine, May 2007 CPU : NEC V850 (uPD70F3716GC, 20MHz) Internal Flash ROM 256KB, SRAM 24KB
I wrote another stupid Linux device driver. :o This time the driver uses netfilter and can catch various conditions of TCP/IP(v4, at this moment) protocol stack. I named it 'holden'. :) It's available here.
Well, I heard Linux TCP/IP stack sometimes sent out TCP segments with both RST and ACK bits are ON, and I wanted to catch such an event. This is why I did this small work. Anyway, it's just a relaxation before going into RFC 793 deeply.
On April 6, I applied a small new software project creation for SourceForge.net and I've been waiting if the application was approved or not for several days.
I know it was Friday April 6 and it would be at most Tuesday (in Japan) or so. But, I found there was an approval mail on my SourceForge account which was sent out to me on April 6(!?). :( Why was not the mail forwarded to me??? But anyway, I'm going to upload the programs I wrote recently like this (or this) and form a small project from this week end. :)
It's 12 weeks and thatsdone 3.0 is about 6cm tall. Oh well, but accoding to my wife, 6cm does not include its legs. Today, she got also a 25min video, and I saw the baby is waving its hands and legs for the first time. It's like dancing. :) I hope this baby can be born without any trouble...
Bibimbap is a popular Korean cuisine in also Japan. It's a favorite dish of my wife, and here is tonight dinner. :o
As the Wikipediabibimbap page says, basically they use beef not pork and a plenty of vegetables. Tonight, my wife arranged simply that her own way, but anyway I like it and we enjoyed. :)
Today, I went to Diamond City Mu again with my wife to buy a present for my mother. We were looking for a photo stand so that she can put photos of her grand child due in this November. :) But, it's not the point of today. When I went to Harvest Village, I heard Hanamizuki of Yo Hitoto. But, actually it's not Yo Hitoto herself, but other singer was singing the song in bossa nova arrange.
After hanging around the shopping center including HMV, we found it's Sotte Bosse (http://www.sottebosse.com/). At this moment, they have 2 CDs and all the songs are cover versions of popular Japanese pop songs. I think it's a healing arrange of the original songs in a sense. And so, I bought "innocent view" above. :)
I found there was one April Fool RFC this year after 5 days delay. :)
http://www.ietf.org/rfc/rfc4824.txt
It's a new encapsulation of IP protocol on top of a very traditional link layer. :) Here is an excerpt from the clause 3.5 of the RFC.
|3.5. Protocol Limitations | | Due to the physical characteristics of the transfer channel, bit | error rates are expected to be in the range of 1e-3 (boy scout) to | 1e-4 (professional sailor), and also depend a number of physical | factors. Poor visibility due to weather conditions or lack of | illumination (e.g., night time) can drastically increase the error | rate.
Oh, well. I believe some African people can provide us extremely good quality transmission. Perhaps, error rate 1e-6 and extremely long distance? :o)
| IP-SFS provides no means to handle frame reordering or dual | (multiple) frame reception. Thus, the protocol is not suitable in | environments where interfaces are moving fast and/or when the path of | light is long.
In Japan, about 1800 years ago, there was a genius who can listen to 7 people simultaneously, and so probably he can be a multiplexor among people using IP over SFS. :)
I wrote a stupid Linux device driver for trouble shooting. :o It's available here.
An excerpt from README.txt --- This small module, rye.ko, tries to catch a specific in-kernel event.
At this moment the only supported event is a phenomenon called CSS (Context Switch Storm) [1, 2, 3]. rye.ko creates a kernel thread on insertion, and the thread monitors accumulated number of context switches periodically and tests if context switches rate (per second) gets higher than a threshold given on the insertion time periodically. If the rate exceeded the given threshold, it calls panic(). Thus, using rye.ko with your favorite crash dump utility such as kdump, you can take good information for your investigation. --- I hope trouble shooting guys would enjoy with this module. :o
As I don't like noisy places, I like the atmosphere of Syuri-ten. :) One more interesting thing about the restaurant is that there are many foreign guests there because it's in Ebisu.
Its name is Everly, and their official home page is here.
They look like just a classic quartet in the above photo, but their uniqueness is that they play both classic and popular music. I mean they play also popular instruments such as guitars and sing songs not only classical instruments. As it was an open air concert, I'm not so sure their tune is very good or not. But, they have good atmosphere and make their audience have fun. So, I want to see their pops concert. :)
BTW, my wife found that two violin players look almost the same. In the closing talk their open air performance, it turned out that actually they were brothers.
Hyakkan (百干) is a good Japanese style dining restaurant. There seems to be three branches in Tokyo, and today we had a dinner at Hyakkan Kokubunji.
Their catch phrase is 'aburyanse (あぶりゃんせ)' in Japanese as you can see their homepage. It's a bit difficult to explain what あぶりゃんせ means in English because we Japanese love fishes very much and there are several differences in cooking method of fishes between western world and Japan, but anyway, roughly speaking that means something like "Let's broil (a delicious fish and enjoy it) ."
Today, my wife had a broiled Atka mackerel (Hokke, ほっけ in Japanese) and I had a boiled sea bream. (On the menu, it's Medai(目鯛) but I'm not sure its English name exactly.) Very good. :)
The right one above is my wife's dessert, a citron sherbet. :)
In Japan, April is the season of Cherry Blossoms. We really love the flower. :)
There is Koganei Kouen (Koganei park, 小金井公園) about 15 minutes walk from my room, and it's well known for its Cherry Blossom in Tokyo. I and my wife went there for the first time since we moved here, Kodaira city. I guess today was the best day this year for enjoying at Koganei Park. Here are several pictures I took there.
As you can see above, thousand of people are enjoying their parties seeing beautiful cherry blossoms. :) I heard that it's banned drinking in a public place such as a park in the USA. How poor people they are... :o
BTW, there are terrible traffic jams around Koganei Park especially along Itsukaichi-Kaido avenue (五日市街道) in this season. Today, of course there are. :o As for us, we walked to Koganei Park, so no problem except backache. :o
Recently, at work I'm using a Linux kernel built-in proiler, oprofile.I didn't know so far several features like the followings.
It's possible to collect system wide symbol information including even JavaVM by specifying just like the below. I mean without any binary image files. # opreport -lc
It's not simple to collect symbol information of loadable kerel modules. :(
Anyway, oprofile is a really useful tool because in the first place it's a kernel built-in utility. That means we do not need to re-compile nor modify the target kernel. :)
Here are my seeveral things to do.
Try other CPU events than timer such as TLB miss.
Establish a how-to to collect lkm (loadable kenel module) information.
Finish writing up analysis howto for my colleagues.
Finish writing up analysis scripts of oprofile (human readable) log files.
etc.
BTW, actually I wrote two analysis scripts on my way back home Today in the train. I'm wondering whether I should make it public as GPL2 like this one or not because I spent no woking time... :o
Here are several useful links (for Japanese users).
Today, I went to the New National Theatre Tokyo to see "La Forza del Destino" ("運命の力" in Japanese) with my wife. The theatre is located in Hatsudai, one station from Shinjuku station via Keio-Shinsen line (a subway line). Please note that the tower in the left picture below is not a part of the theatre although its name is Tokyo Opera City. Actually, they were built in collaboration with the goverment and nongovermental organizations.
As you can see in the second and the third pictures, the theatre is contemporary and very good one.
EXCEPT!
Their seats.
Compared to seats of other new theatres in Tokyo, their cushion is not good. Both of us felt pain on our hips. :( Well, let's get down to the main subject. :)
Originally, my father-in-law was to see this performance with my wife. But, because of a family stuff he could not make it and I was his replacement. :o It's my first experience to see a classic opera and so at first I afraid I could feel sleepy because it's spoken (sung?) in Italian completely. But, he generously sent us a DVD and we had a time to learn the story line. One more thing which made me surprised was Japanese opera theatres has a big display for subtitle in Japanese. :o Thus, we could enjoy the performance well. :)
Higo-monzu (ひごもんず) is indeed a Japanese name, but it's a Kumamoto dialect. Higo means Kumamoto, but I'm not sure what 'monzu' means. I guess a Kumamoto guy or something like that.
Attribute of Kumamoto style ramen is its tonkotsu (bones of pork) based soup like other Kyushu area ramen, and that means the soup has plenty of body. The above photo is a ramen (chinese noodle) with boild pork chop. :)
Among aliens there, Ferengi would be one of the most impressive races. They are known as the best merchants in in the alpha quadrant of the Galactic System.
Their strength in earning money comes from their philosophy, or so called the Rules of Acquisiton.
Reading both of them, I found there are several differences of nuance, and I felt a bit Ferengi philosophy is a mixture of existing ones in the world. But, they are not the points today.
Here are my favorite rules.
#7 Keep your ears open.
#60 Keep your lies consistent.
#162 Even in the worst of times, someone turns a profit.
(Unwritten rule) When no appropriate rule applies, make one up.
Esepecially, the second and the last. :o Note that #7 is saying "ears" not "eyes." I believe you can understand why looking at Ferengi people physical appearance. :)
BTW, I didn't know there were several books about Ferengi and their rules of acquisiton. For example, the below. I want it. :o "The Ferengi Rules of Acquisiton" ISBN 978-0671529369
Well, this is another nostalgia. In the middle of 80's, there was a very popular TV show called Yuyake-Nyan-Nyan (夕やけニャンニャン).
Psychic Magic of G.I. Orange was its ending theme tune, and so almost every week day we heard this tune in the evening. Most of the people there were high school girls in those days. But, now they are in their late 30s. :o
http://www.youtube.com/watch?v=_9501LNJk3Y
Lyrics are here. http://www.mp3lyrics.org/g/gi-orange/psychic/ In those days, I felt the song is cool. But now, I'm not sure why I did so...
At work, I need to visualize various quantities such as CPU usage. So, GNUPLOT is a really useful tool for me. I'm not sure how many scripts I wrote since last year. :)
So far, I've been using GNUPLOT mainly generating time-series graphs because it's difficult creating (stacked) bar graphs like the one below. Thus, I had to use GNUPLOT and other graph plotting library such as JFreeChart. :(
BUT!
I found GNUPLOT 4.2 finally supported stacked bar graphs. Actually the below one is an example on the demo page of GNUPLOT 4.2. :)
Finally, I can say good-by to JFreeChat. :)
Well, please note that I'm not saying JFreeChart is not cool, but the tool I'm using at work which uses JFreeChart is rotten.
My father also had a Subaru 360 about 35 years ago, and I remember well that I played with the car when he's at work during my early childhood. My family really loved the car. :)
I'm not sure when I saw a Subaru 360 last time on the road. But, even at that time, I saw it from away. So, I didn't recognize the actual size of Subaru 360. Today, it was about 2 or 3 meters and the car was parking. I was surprised realizing how small it is. It's like a Mini-Cooper, but actually in Japan Subaru 360 had a petname, a ladybug.
That reminded me of happy days with my father long long ago. :)
As you can see in the source code, this is a straight forward, SIGNAL based, multi-process model and stupid program. :o But, anyway it works at least for me. I wrote a similar test program about 10 years ago for an IBM compatible general purpose mainframe computer in assembly language when I was a freshman.
Well, time goes by, and there is almost no progress. :(
Note: Some folks might know my employer and what I'm doing at work. But, this very small work was done completely using my spare time. So, please do NOT blame me.