Sunday, May 20, 2007

Musashino Sabo

Today, I went to Diamond City Mu with my wife again.
She really loves shopping. :o

We had a lunch at Musashino Sabo on the first floor there,
and my choice was Carbonara with Kyoto style bean paste(Miso) cream.



At first, I was not sure if bean paste matches with pasta, but flavor of
bean paste was not so strong and I like it. :)

Saturday, May 19, 2007

3 People Please!?

I was astonished by my wife today.

We wanted to have a dinner at an Italian restaurant near our house, Pasta and Pizza Primo Piatto, and went there around 18:30.

My wife said to a floor lady in the entrance there, "Three people, please."

Yes, our child is indeed due in the end of this October about five months future, but at this moment we need only two seats. :o

Well, what are you thinking about, honey? :o)

Haguregumo



Haguregumo (浮浪雲) is a bit new Chinese noodle (Ramem) restaurant around our room, Koganei city, Tokyo.
Their noodle is a bit thick, and we have 3 x 4 = 12 choices for the recipe.
  • Soup ---- strong, normal, thin
  • Noodle -- hard,normal,soft
  • Oil------- much,normal, less, none
I chose normal soup, hard noodle, normal oil, and normal soup turned out to be the correct choice for me. I like it. :)

BTW, I don't know what is "House Ramen" (家系ラーメン), but according to literature on Internet, there seems to be very deep world. I should consult with a professional Ramen guy.
Hi, Harada-san, are you reading?
Give me a lecture, please. :)

Friday, May 18, 2007

Google Analytics

Recently, I got a Google Analytics account.
Especially, I like the following geological report. :)



Note that my mouse pointer was on Australia when I captured the above screenshot.

Thursday, May 17, 2007

A Bug Report

Today, I got a bug report of this device driver.

'rye' had two bugs. :(
  1. CPU usage rate was different from its specification. :(
  2. There was a bug in saving the current sampling data of CPU usage data. So, calculated CPU usage rate decreased at ever sampling time even under a stable work load in case of an SMP machine. :(
Now, rye is version 3.0 at this URL.

Thanks, Furuta-kun! :)

Tuesday, May 15, 2007

Oyster Bar Again

Again, I had a lunch at the Grand Central Oyster Bar & Restaurant, Tokyo today. :)



My choice was Risotto with Oyster.

Actually, last time I went to the restaurant it was to be a small welcome party for one of my colleagues. But, unfortunately the guy was not good recently and we went to the restaurant without him cause we cannot wait for his recovery. :o

Today, finally we could welcome him.

Thanks Ushida-san, again! :)

Monday, May 14, 2007

Wikipedian

I got an account of Wikipedia.
I just wanted to correct a typo of an article about object-oriented programming, but something pushed me.
Here is my userpage at Wikipedia, and I seem to be a new Wikipedian from Today. :)

Sunday, May 13, 2007

Linux Context Switch



Since I wrote this entry, I was wondering about the top address of a stack
trace of a process sleeping in schedule() like below:
crash> bt
PID: 2086 TASK: ffff81007f108080 CPU: 1 COMMAND: "sendmail"
#0 [ffff810078291950] schedule at ffffffff80260ab2 <== Here!
#1 [ffff810078291a30] schedule_timeout at ffffffff80261416
#2 [ffff810078291a80] do_select at ffffffff80211369
[snip]

crash> rd ffff810078291950 2
ffff810078291950: ffff810078291a28 0000000000000031 (.)x....1.......

My point is that '0xffffffff80260ab2' is not equal to the value stored
at '0xffff810078291950' in the above case. This is different from other
stack frames. Actually, in the above example values stored at the stack
frame #1 and #2 are exactly the return addresses at the rightmost column
of each line.

Q1.

First of all, what does the address '0xffffffff80260ab2' mean?
Assuming the address is a return address of a FAR call instruction,
I disassembled 0xffffffff80260ab2 - five bytes, and got the following:


crash> dis ffffffff80260aad 16
0xffffffff80260aad : callq 0xffffffff80262898 <__kprobes_text_start>
0xffffffff80260ab2 : mov %gs:0x0,%rsi
[snip]

As shown in the above, its procedure name is 'thread_return' presumably
in schedule(). Here, schedule() is the central function to make decisions
which process to run.

Q2.

Next question is why 'thread_return?'

I looked into crash-4.0-2.13 source tree and understood the reason.
Here is a call graph of 'bt' command.

bt_cmd (./kernel.c:1351 or ./kernel.c:1359)                    
back_trace ()
mach_dep->get_stack_frame (./kernel.c:1518)
x86_64_get_stack_frame (x86_64.c:2296)
x86_64_get_pc (x86_64.c:2510)
# Here, if thread_struct does not have %rip as its member,
# x86_64_get_pc returns 'thread_return' as %rip.
# That's the case of kernel 2.6.11 or later.

In short, 'bt' command of crash always returns the same address for a process
which is NOT on a physical CPU.
The stack frame #0 above is the place where the process left in-execution
state and switched to the next process to get the CPU.
In short, a context switch occurs. So, the return address must be
the instruction pointer (%rip) when the previous/released CPU
process gets CPU and resume execution again.
As it's natural that there are several things to do for kernel after switching
one process to another, thus the return address of can be different from
usual function call cases.

Q3.

My third question is why there is no 'thread_return' address on the top of
stack frame #0?

We can understand the reason loooking into schedule().
Here is another call graph of schedule().
schedule()
context_switch() # an inline architecutre independent C function
switch_to() # a macro in assembly language
callq __switch_to # an architecture dependent C function
Actually, 'switch_to' is defined like the following in case of x86_64
(include/asm-x86_64/system.h), and as is shown below the stack pointer
%rsp is switched to the next process's one before calling '__switch_to.'
 23 #define switch_to(prev,next,last)                                      \
24 asm volatile(SAVE_CONTEXT \
25 "movq %%rsp,%P[threadrsp](%[prev])\n\t" /* save RSP */ \
26 "movq %P[threadrsp](%[next]),%%rsp\n\t" /* restore RSP */ \
27 "call __switch_to\n\t" \
28 ".globl thread_return\n" \
29 "thread_return:\n\t" \
30 "movq %%gs:%P[pda_pcurrent],%%rsi\n\t" \
31 "movq %P[thread_info](%%rsi),%%r8\n\t" \
32 LOCK_PREFIX "btr %[tif_fork],%P[ti_flags](%%r8)\n\t" \
33 "movq %%rax,%%rdi\n\t" \
34 "jc ret_from_fork\n\t" \
35 RESTORE_CONTEXT \
[snip]

The call instruction on line 27 above does not save its return address
('thread_return') into the previous stack frame top, because the kernel
changed the current stack frame from the previous one (saved RSP on line 25)
to the next (restored RSP on line 26) just before that.

Thus, thread_return does not appear in the stack frame of the previos process.

Q4.

The last question is what is 'thread_return'?
But, the name looks like self-explaining, and actually on the line 28
of 'switch_to' definition above there it is. It's just the re-entry
point for a process on returning from '__switch_to().'

I hope this article is useful for people.

Saturday, May 12, 2007

Objet

Yesterday, I went to this place near the JR Shin-Narashino Station to attend at a kind of business camp.

On my way to the place from the station, I encountered a strange object.
What is this at all? :o

Thursday, May 10, 2007

Oyster Bar de Lunch

I had a gorgeous lunch today at this restaurant.
It's almost two years since the last time I went there.

I really love the new england cram chowder below.



The main dish was an oyster bowl below left.



Thanks, Ushida-san! :)

BTW, I know "Lunch de Oyster Bar" is correct, but it's actually a Japanese phrase meaning "Lunch at Oyster Bar." :o

Monday, May 07, 2007

Improving Back Trace

After writing this entry, I was thinking how we can improve 'bt' command of the crash utility.

I wrote an LDAS (Linux Dump Analysis Script) of Alicia, btx.ldas, as an alternative of 'bt'. (btx.ldas is available at this URL as usual.)

Roughly speaking, btx.ldas is a filter of 'bt' command results.
The basic idea is that there are return addresses on the right most colums and that means there should be call instructions just before the address. Furthermore, it's possible to check whether a supposed return address is really correct can by seeing if the operand of a call instruction is the address of the next stack frame or not.
That's what btx.ldas does basically.
Of couse, there are several technical issues, but as shown in the following, this approach looks to work. :)

The below examples are results by 'btx.ldas' and crash 'bt' command of the same "sendmail" process taken on my PentiumD 930 / FedoraCore 6 (x86_64) box.

(1) crash 'bt' command example

Among the 22 stack frames shown below, there are only 6 correct ones as I wrote in this entry.
btx.ldas checks five bytes (size of a call instruction) before the rightmost addresses colored in blue in each stack frame.

alicia> bt 2086
PID: 2086 TASK: ffff81007f108080 CPU: 0 COMMAND: "sendmail"
#0 [ffff810078291950] schedule at ffffffff80260ab2
#1 [ffff810078291a30] schedule_timeout at ffffffff80261416
#2 [ffff810078291a80] do_select at ffffffff80211369
#3 [ffff810078291b70] proc_alloc_inode at ffffffff802f48dc
#4 [ffff810078291b80] alloc_inode at ffffffff80225746
#5 [ffff810078291bb0] inotify_d_instantiate at ffffffff802e6a70
#6 [ffff810078291bd0] d_rehash at ffffffff80240566
#7 [ffff810078291bf0] proc_lookup at ffffffff802277cd
#8 [ffff810078291c20] proc_root_lookup at ffffffff80254ddd
#9 [ffff810078291c40] do_lookup at ffffffff8020cade
#10 [ffff810078291c70] dput at ffffffff8020cff4
#11 [ffff810078291c90] __link_path_walk at ffffffff8020a196
#12 [ffff810078291cf0] vsnprintf at ffffffff80219e0f
#13 [ffff810078291d00] link_path_walk at ffffffff8020e776

#14 [ffff810078291d90] core_sys_select at ffffffff802d8dd8
#15 [ffff810078291e00] __next_cpu at ffffffff8033bf39
#16 [ffff810078291e10] nr_running at ffffffff802861c5
#17 [ffff810078291e30] loadavg_read_proc at ffffffff802f7be7
#18 [ffff810078291e60] lock_kernel at ffffffff802627fb
#19 [ffff810078291e80] de_put at ffffffff802f480b

#20 [ffff810078291f20] sys_select at ffffffff80216185
#21 [ffff810078291f80] system_call at ffffffff8025c00e
RIP: 00002aaaac4cdb93 RSP: 00007fff27a8fcb8 RFLAGS: 00010206
RAX: 0000000000000017 RBX: ffffffff8025c00e RCX: 000000009999999a
RDX: 0000000000000000 RSI: 00007fff27a90580 RDI: 0000000000000005
RBP: 00007fff27a90580 R8: 00007fff27a90600 R9: 00007fff27a8eadf
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000004
R13: 0000000000000001 R14: 0000000000000001 R15: 0000555555816330
ORIG_RAX: 0000000000000017 CS: 0033 SS: 002b

(2) btx.ldas result

Below is an example using btx.ldas and Alicia-1.1.5.
Extra 16 frames above were suppressed.

alicia> load 'btx.ldas';
alicia> btx 2086;
PID: 2086 TASK: ffff81007f108080 CPU: 0 COMMAND: "sendmail"
#0 [ffff810078291950] schedule at ffffffff80260ab2
#1 [ffff810078291a30] schedule_timeout at ffffffff80261416
#2 [ffff810078291a80] do_select at ffffffff80211369
#3 [ffff810078291d90] core_sys_select at ffffffff802d8dd8
#4 [ffff810078291f20] sys_select at ffffffff80216185
#5 [ffff810078291f80] system_call at ffffffff8025c00e
RIP: 00002aaaac4cdb93 RSP: 00007fff27a8fcb8 RFLAGS: 00010202
RAX: 0000000000000017 RBX: ffffffff8025c00e RCX: 000000001eb851ec
RDX: 0000000000000000 RSI: 00007fff27a90580 RDI: 0000000000000005
RBP: 00007fff27a90580 R8: 00007fff27a90600 R9: 00007fff27a8eadf
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000004
R13: 0000000000000001 R14: 0000000000000001 R15: 0000555555816330
ORIG_RAX: 0000000000000017 CS: 0033 SS: 002b

Global Optimization

Recently I read an interesting article about a difference of business
structures between western countries and Japan. The article was an interview
of a CRM software company selling president.

The essential point of the interview for me was like the following:

In order to exploit CRM software enough, a company must optimize its
business activities company wide not per business division basis.
For example, a design center of its products and factories must cooperate
shooting for the maximum benefit of the company not each business division.
Quite a lot of Japanese companies have a problem from that point of view.
Simply speaking, it's a sectionalism among business divisions in one company.

The president talked about the strategy and foresight of his company more, but
for me the above was interesting.
Actually, that story reminded me of what an executive of my company said
at a kickoff meeting of this fiscal year about a month ago.
I'm working for a company with a quite traditional business structure, and
a lot of business divisions inside the company are always competing with
with each other for their own budgets and benefits. It's like a bureau of
something. :(

BUT!

The executive finally began to say, "Think company wide optimization,
not per business unit basis."
Furthermore, about two weeks later of the kickoff meeting, I heard a news that
the executive would be the president of my company soon. :o

The above story also reminded me of a similar issue I wrote before.
I think a big company can be regarded as a society or a community which
consists of multiple organizations with different stakes. Here,
the organizations are like species with different *genes* carriers.
If the genes are essentially different and competing with each other,
the theory of evolution says that it's not stable that those species
cooperate each other. Thus, the problem is that if employees of my company
are sharing the same gene or not.
I joined my company about three years ago after 10 years of life at other
company, and for me their mentality looks like almost uniform one.
So, I believe the competing divisions share the same gene. :)

BTW, the executive of my company is Mr. Yamashita.
Hey, Yamashita-san, I'm really looking forward to seeing your strong
leadership. :)

Sunday, May 06, 2007

Second Life

Humm....

I got an account of Second Life today.
But, its client software hangs up almost every time... :(

What's wrong!?

Saturday, May 05, 2007

Two Years

It's the second wedding anniversary of us today.

As we had a yakiniku party yesterday, we celebrated the anniversary by a small cake and tea party. :)



We bought them at a cake shop "L'ATELIER DE T" in Toyohashi city.
http://home-eat.seesaa.net/article/33427197.html
http://www.planets.co.jp/select/file16.phtml

The owner chef seems to have worked as the chief pâtissier of the Hotel Okura, Hamamatsu.
My choice was the lemon tarte above, and I like it. My mother in law also loved her Mont Blanc. But, surprisingly enough for me, my wife and my father in law didn't like their cakes.

BTW, when we stayed at my home town, my mother also celebrated us by a small cake party.
My choice was a blue berry tarte below. :)



"Kashiharu"
http://www.kashiharu.com/acce06.htm

Friday, May 04, 2007

Yakiniku Restaurant Hanada

Today's dinner was a Yakiniku party (barbecue party) . :)
It's almost 3 years since I went to a yakiniku restaurant.



The restaurant seems to be very famous and they indeed offer very good meats. :)
Here are several articles about the restaurant, Yakiniku Restaurant Hanada.
  • http://www.jangsanet.com/2-yakiniku/mikawa/toyohashi/hanada/hanada.htm
  • http://r.tabelog.com/aichi/rstdtl/23001167/
BTW, about 4 years ago, I went to this yakiniku restaurant almost once a week. :o
I really loved the restaurant, but now I live very far from the place.

BTW, one of my former colleague is planning to have a party at the restaurant.
Hey, Saeki-kun, it's YOU! Are you reading?
I'm looking forward to seeing you guys and eating together there. :)

Nameshi Dengaku

Today's lunch, Nameshi Dengaku (菜めし田楽) at Kiku-So (きく宗) in Toyohashi, Japan. :)
I like it. :)



Nameshi Dengaku is a kind of Miso Dengaku.

Dengaku is a category of Japanese traditional food. http://ja.wikipedia.org/wiki/%E5%91%B3%E5%99%8C%E7%94%B0%E6%A5%BD(Japanese)
Miso is soybean paste.

"Nameshi" can be decomposed into "Na" and "Meshi". Here, "Na" means leafs of Japanese radish, and "Meshi" is rice.

Thus, "Nameshi Dengaku" can be translated like "Dengaku with soybean paste and Rice with Japanese radish".

BTW, Dengaku is a kind of Japanese Festivals.
http://en.wikipedia.org/wiki/Dengaku (English)
http://ja.wikipedia.org/wiki/%E7%94%B0%E6%A5%BD (Japanese)
According to the Japanese Wikipedia page above, the name comes from the festival name because skewers of (Miso) Dengaku looks like the foot gear used in Dengaku festival.

Wednesday, May 02, 2007

Shimada Rose Hill Garden

Shimada city is located in the middle of Shizuoka prefecture, Japan. It faces to the Pacific Ocean, so they have very calm climate and it's a very good place for growing flowers like roses.



"島田市ばらの丘公園" (Shimada-si Bara no Oka Kouen) is a kind of flower park dedicated for roses. They don't have an english organization name, but "Shimada City Rose Hill Garden" would be appropriate, I think.



My mother loves the garden, and she often goes to the place when she got guests.
This time, it's May 2 we went there. Unfortunatelly, it's a bit earlier than the peak of this season. But, there were plenty of beautiful roses in the greenhouse. :)

Moto-chan no Itai Hanashi



Moto-chan no Itai Hanashi (もとちゃんの痛い話)
ISBN 4-04-160010-3
Motoko Arai (新井素子)

Well, there is no English translation of this book. In English literal translation, the title would be something like "Moto-chan's Sore Story."
Motoko Arai is a famous Sci-Fi/Fantasy writer in Japan, and this is her essay when she got a painful disease. As usual her writing style, it's not a mournful hospital diary but a good light essay, so we can share her various interesting thoughts.

Actually, when I went back to my home town in this Golden Week vacation, I found my mother was working on the book, and I was also captured. :o

Monday, April 30, 2007

Yokohama-ya

Yokohama-ya (横濱屋) is a Ramen (Chinese Noodle) restaurant chain. They have about 15 restaurants in mainly Tokyo and Kanagawa. In this sense, Yokohama-ya is different from small but good stand-alone Ramen restaurants along Koganei Kaido avenue.

My wife and I had already finished evacuating our fridge today before our trip to our home towns from tomorrow, thus tonight dinner was Ramen, again. :o

Here is my choice.

Chives and Beansprouts Ramen with sliced pork.
I felt it's about 1.5 starts. :(

Sunday, April 29, 2007

Interface Magagine '07/06

I bought '07/06 issue of the Interface Magagine.
http://www.cqpub.co.jp/interface/
http://www.cqpub.co.jp/interface/contents/2007/200706.htm

Yes, it's the magagine that gave me an NEC V850 CPU board. :)

When I read the previous issue in detail last month, I felt it's too elementary because example programs are:
  • Blinking an LED on the board
  • Sending a "Hello, World!" message infinitely
But, there are practical application examples such as a line-tracing car controlled by the V850 board in this month issue, and finally that makes sense for me. Maybe the editors were too busy for preparing the giveaway. :o

BTW, one of the article was about a u-ITRON (TOPPERS) port for the board.
I'm wondering if I should go ahead for another program monitor...
Hey, Furukawa-san, are you reading?
What do you think about? :)

Smetana

Today, we had a dinner at this Russian restaurant, Smetana (Сметана, スメターナ).


Appetizer.



The drink in white is a glass of Kumis (Кумыс) .
http://en.wikipedia.org/wiki/Kumis
http://ru.wikipedia.org/wiki/%D0%9A%D1%83%D0%BC%D1%8B%D1%81



Borsch



The main dish, Kievski escallop.
Also a glass of Nemiroff Honey Pepper Ukrainian Vodka. :)

I like the restaurant, but my wife says she prefers to Sungari in Shinjuku, Tokyo.

Friday, April 27, 2007

14 Weeks


In Japan, a big vacation season is beginning from tomorrow, and my wife had a medical check before the vacation.

Now, thatsdone 3.0 is 14 weeks and 16cm tall without any troubles. :)

Today, the baby seemed to be sleeping and kept still at first, but after a few minutes of the check, woke up and was waving its hands and legs.
Well, I'm not sure if the baby was saying, "Don't wake me up!" :o)

Thursday, April 26, 2007

A Farewell Party

In Japan, usually March and September are farewell party seasons. So, in the beginning of this April, there was a organization restructuring at work and I had several farewell and welcome parties as usual.
But, last week I got a bit shocking news that a colleague of mine would leave our organization by the end of this April. I'm not sure why it's in the end of April not March. Could be because of his new organization's convenience. Anyway, I hope he and his family live in happy harmony there, his wife's home town and also close to his home town. But, what I'm wondering is how the remaining people can fill the role which he did for these two years.

Well, the party was held at Inataya (稲田屋), Shinagawa.
http://www.inataya.co.jp/store/sinagawa/index.html



It's an average class Japanese style pub, and I drank a bit too much Beer and Japanese Sake there. So, I fell over after leaving the pub, and what is worse, I hit my legs hard with something. My legs are still aching. :(
Two my bosses witnessed that. :o

BTW, I didn't know it's "Inata-ya" not "Inada-ya." :o

Monday, April 23, 2007

Linux Kernel Linkage

I have to confess that I didn't know Linux kernel (general?) linkage convention well till the last Sunday. :(

I've been feeling strange we often get unknown function call entries in
kernel stack traces like the below example when we are running Linux on
x86 or x86_64 architecture. Among the stack trace below, entries in red
cannot be found even if you looked into the source code in detail.

crash> bt
PID: 2115 TASK: ffff81007f0ff7d0 CPU: 0 COMMAND: "sendmail"
#0 [ffff810078269950] schedule at ffffffff80260ab2
#1 [ffff810078269a30] schedule_timeout at ffffffff80261416
#2 [ffff810078269a80] do_select at ffffffff80211369
#3 [ffff810078269b70] proc_alloc_inode at ffffffff802f48dc
#4 [ffff810078269b80] alloc_inode at ffffffff80225746
#5 [ffff810078269bb0] inotify_d_instantiate at ffffffff802e6a70
#6 [ffff810078269bd0] d_rehash at ffffffff80240566
#7 [ffff810078269bf0] proc_lookup at ffffffff802277cd
#8 [ffff810078269c20] proc_root_lookup at ffffffff80254ddd
#9 [ffff810078269c40] do_lookup at ffffffff8020cade
#10 [ffff810078269c70] dput at ffffffff8020cff4
#11 [ffff810078269c90] __link_path_walk at ffffffff8020a196
#12 [ffff810078269cf0] vsnprintf at ffffffff80219e0f
#13 [ffff810078269d00] link_path_walk at ffffffff8020e776

#14 [ffff810078269d90] core_sys_select at ffffffff802d8dd8
#15 [ffff810078269e00] __next_cpu at ffffffff8033bf39
#16 [ffff810078269e10] nr_running at ffffffff802861c5
#17 [ffff810078269e30] loadavg_read_proc at ffffffff802f7be7
#18 [ffff810078269e60] lock_kernel at ffffffff802627fb
#19 [ffff810078269e80] de_put at ffffffff802f480b

#20 [ffff810078269f20] sys_select at ffffffff80216185
#21 [ffff810078269f80] system_call at ffffffff8025c00e

Finally, I understood the reason.

Below code chunk is excerpted from 'arch/x86_64/kernel/traps.c' of RHEL5 Update 0 kernel (based on linux-2.6.18). The function 'dump_trace' is the one which prints out stack traces, and note that lines from 269 to 272 say something strange. That means 'dump_trace' cannot determine boundaries of stack frames strictly. Thus, taking a stack area as if it's an array of pointers, the function prints an entry into the stack trace if the value looks like an in-kernel function. :(
242 void dump_trace(struct task_struct *tsk,
struct pt_regs *regs, unsigned long * stack,
243 struct stacktrace_ops *ops, void *data)
244 {
245 const unsigned cpu = smp_processor_id();
246 unsigned long *irqstack_end =
(unsigned long *)cpu_pda(cpu)->irqstackptr;
247 unsigned used = 0;
248
[snip]
259 /*
260 * Print function call entries within a stack. 'cond' is the
261 * "end of stackframe" condition, that the 'stack++'
262 * iteration will eventually trigger.
263 */
264 #define HANDLE_STACK(cond) \
265 do while (cond) { \
266 unsigned long addr = *stack++; \
267 if (kernel_text_address(addr)) { \
268 /* \
269 * If the address is either in the text segment of the \
270 * kernel, or in the region which contains vmalloc'ed \
271 * memory, it *may* be the address of a calling \
272 * routine; if so, print it so that someone tracing \
273 * down the cause of the crash will be able to figure \
274 * out the call path that was taken. \
275 */ \
276 ops->address(data, addr); \
277 } \
278 } while (0)
279
280 /*
281 * Print function call entries in all stacks, starting at the
282 * current stack address. If the stacks consist of nested
283 * exceptions
284 */
285 for (;;) {
286 char *id;
287 unsigned long *estack_end;
288 estack_end = in_exception_stack(cpu, (unsigned long)stack,
289 &used, &amp;amp;amp;amp;amp;id);
290
291 if (estack_end) {
292 if (ops->stack(data, id) <>stack(data, "");
296 /*
297 * We link to the next stack via the
298 * second-to-last pointer (index -2 to end) in the
299 * exception stack:
300 */
301 stack = (unsigned long *) estack_end[-2];
302 continue;
303 }

304 if (irqstack_end) {
[snip]
323 }
324 break;
325 }
The reason comes from the linkage convention of Linux (x86 and x86_64, at least).

In those architectures, callee functions do NOT save any registers except ones which the callees destroy. Even the stack pointer (%sp or %rsp register). Only return addresses are saved automatically by 'call' instructions.

That's why, different from other architectures/operating systems like SPARC/Solaris, it's difficult to track back the caller functions correctly among the stack area in case of Linux/x86(x86_64). :(

Of couse, this must come from performance consideration. But still, it makes trouble shootings difficult, I think. :(

Friday, April 20, 2007

Linux Function Call Catcher

I wrote another stupid device driver, 'allie.'

'allie' uses kprobe which is a built-in feature in RHEL4 or later.
If you want to know call trees of a particular in-kernel function such as 'ip6_xmit()' but don't want to recompile the entire kernel, 'allie' will help you very much.
It's available here under GPL2 as usual. :)

When you want to know function call graphs of a particular in-kernel function, for example, 'mpage_bio_submit', then you can see them by installing 'allie' with insmod parameters like the following:

# insmod allie.ko symbol= mpage_bio_submit count=3

Here are several notes.
  1. Lines in red below are kprobe and allie origin trace entries.
  2. Under Linux(x86 and x86_64) environment, we often see actually unrelated function call entries because of Linux linkage convention as I wrote here. Don't be confused.
  3. count=3 means to show only 3 stack traces because in-kernel functions are usually called very frequently and your /var/log/messages could be full.
  4. kprobe uses 'int3', thus we cannot catch function calls under interrupt disabled state.
The below example is excerpted from a log file included in allie-1.0.tar.gz.
-----
Apr 19 20:06:59 rhel5 syslogd 1.4.1: restart.
[snip]
Apr 19 20:47:14 rhel5 kernel: allie_init: called
Apr 19 20:47:14 rhel5 kernel: allie_init: flags=00000000, action=1, count=3, verbose=0
Apr 19 20:47:14 rhel5 kernel: allie_init: symbol=mpage_bio_submit
Apr 19 20:47:42 rhel5 kernel: [] allie_pre_handler+0x3b/0x61 [allie]
Apr 19 20:47:42 rhel5 kernel: [] kprobe_exceptions_notify+0x187/0x3dc
Apr 19 20:47:42 rhel5 kernel: [] notifier_call_chain+0x19/0x29
Apr 19 20:47:42 rhel5 kernel: [] do_int3+0x39/0x6a
Apr 19 20:47:42 rhel5 kernel: [] int3+0x1e/0x24

Apr 19 20:47:42 rhel5 kernel: [] mpage_bio_submit+0x1/0x1d
Apr 19 20:47:42 rhel5 kernel: [] mpage_readpages+0xec/0xf9
Apr 19 20:47:42 rhel5 kernel: [] get_page_from_freelist+0x96/0x310
Apr 19 20:47:42 rhel5 kernel: [] common_interrupt+0x1a/0x20
Apr 19 20:47:42 rhel5 kernel: [] ext3_readpages+0x0/0x15 [ext3]
Apr 19 20:47:42 rhel5 kernel: [] ext3_readpages+0x0/0x15 [ext3]
Apr 19 20:47:42 rhel5 kernel: [] __do_page_cache_readahead+0x11f/0x1c6
Apr 19 20:47:42 rhel5 kernel: [] ext3_get_block+0x0/0xbd [ext3]
Apr 19 20:47:42 rhel5 kernel: [] blockable_page_cache_readahead+0x46/0x99
Apr 19 20:47:42 rhel5 kernel: [] page_cache_readahead+0xb3/0x178
Apr 19 20:47:42 rhel5 kernel: [] do_generic_mapping_read+0x137/0x468
Apr 19 20:47:42 rhel5 kernel: [] __generic_file_aio_read+0x16f/0x1b6
Apr 19 20:47:42 rhel5 kernel: [] file_read_actor+0x0/0xd1
Apr 19 20:47:43 rhel5 kernel: [] generic_file_aio_read+0x3b/0x42
Apr 19 20:47:43 rhel5 kernel: [] do_sync_read+0xb6/0xf1
Apr 19 20:47:43 rhel5 kernel: [] autoremove_wake_function+0x0/0x2d
Apr 19 20:47:43 rhel5 kernel: [] sched_balance_self+0x1bf/0x208
Apr 19 20:47:43 rhel5 kernel: [] do_sync_read+0x0/0xf1
Apr 19 20:47:43 rhel5 kernel: [] vfs_read+0x9f/0x141
Apr 19 20:47:43 rhel5 kernel: [] kernel_read+0x32/0x43
Apr 19 20:47:43 rhel5 kernel: [] prepare_binprm+0xc7/0xcc
Apr 19 20:47:43 rhel5 kernel: [] do_execve+0xf6/0x1f5
Apr 19 20:47:43 rhel5 kernel: [] sys_execve+0x2a/0x4a
Apr 19 20:47:43 rhel5 kernel: [] syscall_call+0x7/0xb

(Updated slightly on April 29, 2007)

Guts Ramen

Guts Ramen is another Chinese noodle restaurant close to my house.
It's closer to my house than Menya Kotaro, and actually I and my wife go to Guts most frequently among Chinese noodle restaurants along Koganei-Kaido avenue. :)

http://tokyo.gourmet.livedoor.com/restaurant/info/18934.html

They offer very good pork bone based soup Chinese noodles and very soft sliced roast porks, and both my wife and I like them. :)
In addition, Guts is unique for its advertising display and chairs. Especially, chairs look like piggs and kids must like them. :)

Here is tonight dinner. :o


Wraplings, and Chinese noodle with sliced roast pork and sliced green onion.

Wednesday, April 18, 2007

NEC V850 CPU Board Power ON

Well, it's about the V850 CPU board I wrote before.
As I'm not good at soldering, I asked one of my colleagues to assemble separated parts including a USB connector, several external pins and a crystal module.
Today, finally the guy gave me completed board, and it's the Power ON ritual(?) day for the V850 board. It worked without any trouble, and you can see its LED on the upper left corner is lighting.
Thanks, Furukawa-san! :)

Now, I'm considering what I should do next, but I guess I'd write a small monitor program.

BTW, I found an excellent report page about the board. (Japanese only, at this moment)
I'm a pure software guy, and so I cannot compete with him. :o


Interface Magazine, May 2007
CPU : NEC V850 (uPD70F3716GC, 20MHz) Internal Flash ROM 256KB, SRAM 24KB


Monday, April 16, 2007

Another Stupid Device Driver

I wrote another stupid Linux device driver. :o
This time the driver uses netfilter and can catch various conditions of TCP/IP(v4, at this moment) protocol stack. I named it 'holden'. :)
It's available here.

Well, I heard Linux TCP/IP stack sometimes sent out TCP segments with both RST and ACK bits are ON, and I wanted to catch such an event. This is why I did this small work.
Anyway, it's just a relaxation before going into RFC 793 deeply.

Wednesday, April 11, 2007

An in-kernel Event Catcher

I updated the stupid device driver for Linux and made it public here.

Now the driver can:
  1. Monitor context switches rate
  2. Monitor CPU consumption rate
and cause a panic, and thus using it with your favorite crash dump utility, you can take good information about your trouble.

Here is a TODO list.
  1. Make it a bit more foolproof.
  2. Add interrupt rate monitoring. (Straight forward)
  3. Add lock contention monitoring, if possible. (I'm not sure.)
  4. Add monitoring features for other performance sensitive quantities. (Any idea?)
  5. Add some more documentation.
Well, it will be a part of the project I mentioned yesterday.

Tuesday, April 10, 2007

SourceForge

On April 6, I applied a small new software project creation for SourceForge.net and I've been waiting if the application was approved or not for several days.

I know it was Friday April 6 and it would be at most Tuesday (in Japan) or so. But, I found there was an approval mail on my SourceForge account which was sent out to me on April 6(!?). :(
Why was not the mail forwarded to me???
But anyway, I'm going to upload the programs I wrote recently like this (or this) and form a small project from this week end. :)

What kind of project?
Well, stay tuned. :o

Monday, April 09, 2007

Debian GNU/Linux 4.0

I didn't know Debian GNU/Linux 4.0 (etch) was released yesterday. Actually, the release was earlier than my expectation. :o

I've worked on a Debian based system for recent 1.5 years, but the project was mothballed, AGAIN! :(

Hey Mr. T, are you reading?
You have to let us make our works be public!
What in the world do you want me to do? :(

12 Weeks



It's 12 weeks and thatsdone 3.0 is about 6cm tall.
Oh well, but accoding to my wife, 6cm does not include its legs.
Today, she got also a 25min video, and I saw the baby is waving its hands and legs for the first time. It's like dancing. :)
I hope this baby can be born without any trouble...

Sunday, April 08, 2007

Bibimbap

Bibimbap is a popular Korean cuisine in also Japan.
It's a favorite dish of my wife, and here is tonight dinner. :o



As the Wikipedia bibimbap page says, basically they use beef not pork and a plenty of vegetables. Tonight, my wife arranged simply that her own way, but anyway I like it and we enjoyed. :)

Saturday, April 07, 2007

The Ultimate Pointing Device


I bought an interesting pointing device for presentations.
That's it!
And, I don't need a laser poiner anymore. :o

BTW, it reminded me of an old Japanese TV anime. If you are in your late 30s or ealy 40s, I believe you can understand why. :)

Sotte Bosse


Today, I went to Diamond City Mu again with my wife to buy a present for my mother.
We were looking for a photo stand so that she can put photos of her grand child due in this November. :) But, it's not the point of today. When I went to Harvest Village, I heard Hanamizuki of Yo Hitoto. But, actually it's not Yo Hitoto herself, but other singer was singing the song in bossa nova arrange.

After hanging around the shopping center including HMV, we found it's Sotte Bosse (http://www.sottebosse.com/). At this moment, they have 2 CDs and all the songs are cover versions of popular Japanese pop songs. I think it's a healing arrange of the original songs in a sense. And so, I bought "innocent view" above. :)

Friday, April 06, 2007

RFC4824

I found there was one April Fool RFC this year after 5 days delay. :)

http://www.ietf.org/rfc/rfc4824.txt

It's a new encapsulation of IP protocol on top of a very traditional link layer. :)
Here is an excerpt from the clause 3.5 of the RFC.

|3.5. Protocol Limitations
|
| Due to the physical characteristics of the transfer channel, bit
| error rates are expected to be in the range of 1e-3 (boy scout) to
| 1e-4 (professional sailor), and also depend a number of physical
| factors. Poor visibility due to weather conditions or lack of
| illumination (e.g., night time) can drastically increase the error
| rate.

Oh, well. I believe some African people can provide us extremely good quality transmission. Perhaps, error rate 1e-6 and extremely long distance? :o)

| IP-SFS provides no means to handle frame reordering or dual
| (multiple) frame reception. Thus, the protocol is not suitable in
| environments where interfaces are moving fast and/or when the path of
| light is long.

In Japan, about 1800 years ago, there was a genius who can listen to 7 people simultaneously, and so probably he can be a multiplexor among people using IP over SFS. :)

Thursday, April 05, 2007

A Stupid Device Driver

I wrote a stupid Linux device driver for trouble shooting. :o
It's available here.

An excerpt from README.txt
---
This small module, rye.ko, tries to catch a specific in-kernel event.

At this moment the only supported event is a phenomenon called CSS
(Context Switch Storm) [1, 2, 3].
rye.ko creates a kernel thread on insertion, and the thread monitors
accumulated number of context switches periodically and tests if context
switches rate (per second) gets higher than a threshold given on the
insertion time periodically.
If the rate exceeded the given threshold, it calls panic(). Thus,
using rye.ko with your favorite crash dump utility such as kdump,
you can take good information for your investigation.
---
I hope trouble shooting guys would enjoy with this module. :o

Guin Saga 113

The Unknown Underworld
ISBN978-4-15-030884-1

113!

At this moment, almost bimonthly pace. :)

11 Weeks



11 weeks and 30grams.
Well, routine helth checkup is next Monday, but my wife consulted a doctor.
thatsdone 3.0 seemd to be waving its hands. :)

Wednesday, April 04, 2007

Syuri-Ten

Syuri-Ten (首里天) is an Okinawa style dining restaurant in Ebisu, Tokyo.

Gourmet Guide : http://r.gnavi.co.jp/a491600/
Syuri-Ten Home Page : http://www.realize-ryukyu.com/suiten/index.html

As I don't like noisy places, I like the atmosphere of Syuri-ten. :)
One more interesting thing about the restaurant is that there are many foreign guests there because it's in Ebisu.


Thanks Sam!

Oops, Xen Summit, Spring 2007

Xen Technical Summit, Spring 2007

https://www.regonline.com/EventInfo.asp?EventId=124479

OOps, it's too late.
I really wanted to be there. :(

Sunday, April 01, 2007

Interface Magagine 2007/05

Today, I bought the Interface Managine, issue 2007/05.

http://www.cqpub.co.jp/interface/
http://www.cqpub.co.jp/interface/contents/2007/200705.htm

What is interesting is, this issue's giveaway is NEC V850 CPU board. :o

Everly

I saw an interesting quartet in Koganei Park.



Its name is Everly, and their official home page is here.

They look like just a classic quartet in the above photo, but their uniqueness is that they play both classic and popular music. I mean they play also popular instruments such as guitars and sing songs not only classical instruments. As it was an open air concert, I'm not so sure their tune is very good or not. But, they have good atmosphere and make their audience have fun. So, I want to see their pops concert. :)

BTW, my wife found that two violin players look almost the same. In the closing talk their open air performance, it turned out that actually they were brothers.

Hyakkan

Hyakkan (百干) is a good Japanese style dining restaurant.
There seems to be three branches in Tokyo, and today we had a dinner at Hyakkan Kokubunji.

Their catch phrase is 'aburyanse (あぶりゃんせ)' in Japanese as you can see their homepage. It's a bit difficult to explain what あぶりゃんせ means in English because we Japanese love fishes very much and there are several differences in cooking method of fishes between western world and Japan, but anyway, roughly speaking that means something like "Let's broil (a delicious fish and enjoy it) ."

Today, my wife had a broiled Atka mackerel (Hokke, ほっけ in Japanese) and I had a boiled sea bream. (On the menu, it's Medai(目鯛) but I'm not sure its English name exactly.) Very good. :)



The right one above is my wife's dessert, a citron sherbet. :)

Koganei Park and Cherry Blossoms

In Japan, April is the season of Cherry Blossoms. We really love the flower. :)

There is Koganei Kouen (Koganei park, 小金井公園) about 15 minutes walk from my room, and it's well known for its Cherry Blossom in Tokyo.
I and my wife went there for the first time since we moved here, Kodaira city. I guess today was the best day this year for enjoying at Koganei Park.
Here are several pictures I took there.



As you can see above, thousand of people are enjoying their parties seeing beautiful cherry blossoms. :) I heard that it's banned drinking in a public place such as a park in the USA. How poor people they are... :o



BTW, there are terrible traffic jams around Koganei Park especially along Itsukaichi-Kaido avenue (五日市街道) in this season. Today, of course there are. :o
As for us, we walked to Koganei Park, so no problem except backache. :o

Saturday, March 31, 2007

FY06 and FY07

In Japan, a Fiscal Year ends on March 31.
And so, here is my FY06 Summary.
  • Personal affairs
    • Lost a daughter
    • Got a next baby
    • Grandma survived
    • Absent at Nova almost one year
  • Business affairs
    • My project mothballed :(
    • Wrote a bunch of various scripts, not C codes
    • Played with Xen somehow, had a bit of fun!
    • Tamura impressive
    • Okky quit
  • IT industory affairs
    • PS3 stalled
    • Vista stalled
    • Oracle attacked Redhat
  • Social affairs
    • Hitoshi Ueki past away
    • Koizumi ran away
    • Did North Korea really succeed their (so-called) atomic experiment?
Also FY07 Prospect (with no classifications)
  • Chaos at home because of the baby around October :o
  • Visit Europe once :)
  • Bunch of fucking trouble shooting... :(
  • An organization will be disintegrated in midair in the end of the next March :o
  • Big earth quake in Tokyo? :(

A Flowchart for Problem Solving

Recently, some guy posted an interesting flowchart for general problem solving on the Tokyo Linux Users Group ML.



http://g2.sauco.net/flowchart.jpg

I believe copyright of the flowchart above belongs to Mauro Sauco.
Great, Mauro! :)

oprofile (2)

Oops, I wrote several mistakes in the previous entry. :(
  • Typo of command line example :(
    • not # opreport -lc, but # opreport -la
  • Collecting loadable kernel module (lkm) information is straight forward :(
    • Just use -p option just like # opreport -la -p /lib/modules/`uname -r`/kernel/
As for the small scripts I wrote for analyzing system behavior in the previous entry, finally I made them public here, 'opanalyze.'

Things To Do:

  • more documentation
  • examples
  • announce to the oprofile community?

Thursday, March 29, 2007

O.K. Corral

It's not a story about a famous film, but a beef barbecue restaurant (Yakiniku restaurant) in Tokyo.

In Japan, it's a farewell season in the end of March, and so we have many farewell parties. Thus, party organizers need to show their sense of humor.

Today, a colleague of mine suggested to have a farewell party at this restaurant. :o
Great...

BTW, my wife is protesting, and what can I do? :o

Wednesday, March 28, 2007

oprofile

Recently, at work I'm using a Linux kernel built-in proiler, oprofile.I didn't know so far several features like the followings.
  1. It's possible to collect system wide symbol information including even JavaVM by specifying just like the below. I mean without any binary image files.
    # opreport -lc
  2. It's not simple to collect symbol information of loadable kerel modules. :(

Anyway, oprofile is a really useful tool because in the first place it's a kernel built-in utility. That means we do not need to re-compile nor modify the target kernel. :)

Here are my seeveral things to do.

  • Try other CPU events than timer such as TLB miss.
  • Establish a how-to to collect lkm (loadable kenel module) information.
  • Finish writing up analysis howto for my colleagues.
  • Finish writing up analysis scripts of oprofile (human readable) log files.
  • etc.

BTW, actually I wrote two analysis scripts on my way back home Today in the train.
I'm wondering whether I should make it public as GPL2 like this one or not because I spent no woking time... :o

Here are several useful links (for Japanese users).

Monday, March 26, 2007

Almost 10 Weeks and EDC Fixed!



It's 9 weeks and 6 days today. According to the doctor, thatsdone 3.0 is quite well. :)
Also finally, the EDC is fixed on October 23, 2007.

Sunday, March 25, 2007

Xen HVM Guests

Finally, I succeeded to run Xen HVM guests on my PentiumD 930 box today.



The above screenshot shows 2 HVM guests and one PVM guest are running simultaneously.

dom0 : Fedora Core6 (x86_64)
domU(1) : Fedora Core6 (x86_64) Para-Virtalization guest (PVM)
domU(2) : Debian GNU/Linux 3.1 (i386) Full-Virtualization guest (HVM)
domU(3) : Windows 2000 (32bit) Full-Virtualization guest (HVM)

The 2 HVM guests seem to be very slow at this moment.
But, anyway finally I can be on the starting line...