Root Can't symlink dalvik-cache on a rooted Kyocera Milano

[tcaF0xd]

After many trials and tribulations, I successfully rooted a Kyocera Milano (Android 2.3.4 GINGERBREAD) using break_setresuid from github. I installed su and Superuser.apk and that works great. I can run programs like SSHDroid and Android Terminal Emulator with full root access.

My problem is that dalvik-cache is eating up my phone's internal memory. I tried to install Link2SD, but it just hangs after I tell it to work with the second FAT16 partition on my SD card. Using top, I can see that Link2SD spawns an "sh" process which is using 80 to 100% CPU. I let it go for about 30 minutes, but it's clearly in some sort of infinite loop. I also notice that when I type su from Android Terminal Emulator, I get the "#" prompt and when I type my first command (such as "id" which shows root or "ls"), it prints the output but then also just hangs, and top shows the same thing - a 100% CPU "sh" command.

So something seems to be borked, although there's no strace or pstack command, so I don't know how to diagnose the "sh" hang.

So then I tried to just manually deal with these huge files in dalvik-cache. I focused on a single Solitaire app at first. I have a /cache partition with about 100MB, so I first tried creating a dalvik-cache folder there, copying over one of the dex files and then making a symbolic link to it in /cache/dalvik-cache. I made sure to match permissions exactly to what it was before I moved it (system:app_53 and chmod). I even chmod'd the symbolic link itself (using -h). But I would reboot and fire up Solitaire and it would crash. When I do logcat, it says:

E/dalvikvm( 696): Can't open dex cache '/data/dalvik-cache/data@app@com.softick.android.solitaire.klondike-1.apk@classes.dex': Permission denied

Next, I tried doing the same thing on my SD card, creating a dalvik-cache folder and symbolically linking over there. Interestingly, even though I did everything exactly the same, this time I could launch Solitaire without problems through the symlink! Yay.... or not. When I reboot, Android deletes the /data/dalvik-cache/...dex symlink and replaces it with a new dex file.

How can I make Android use the existing symbolically linked dex file after reboot?

I'm so close! All of this because of the rubbish tiny internal drive! And why in the world did they give so much to /cache when only 2MB are used!

Any help would be greatly appreciated.

 

[Joelgp83]

Indeed, you are close.

My problem is that dalvik-cache is eating up my phone's internal memory. I tried to install Link2SD, but it just hangs after I tell it to work with the second FAT16 partition on my SD card.

Using Link2SD is the answer. The key is to link it to an EXT2-formated partition on the SD card, not FAT16/32. You can make one using any number of linux tools (like gparted), or if you have CWM custom recovery, use the "partition SD card" options in it. SDparted is another partition tool designed for this job, but I'm unsure if that's a phone app or something you use on a PC.

 

[tcaF0xd]

Indeed, you are close.
Using Link2SD is the answer. The key is to link it to an EXT2-formated partition on the SD card, not FAT16/32. You can make one using any number of linux tools (like gparted), or if you have CWM custom recovery, use the "partition SD card" options in it. SDparted is another partition tool designed for this job, but I'm unsure if that's a phone app or something you use on a PC.

Thanks for the response, but I did try ext2 (created it using gparted) but it still goes into a loop when I select ext2 from the Link2SD option. Something else is causing certain "sh" forks to go into an infinite loop

 

[Joelgp83]

Ok, on most link2SD setups, the ext2 partition is typically the FIRST partition on the card, with the more conventional FAT32 partition being second. Try that.

Finally, it might be worth it to get in contact with the link2SD developer and explain your problem and findings. It may be that there's a bug in the app he needs to fix, or simply that support for that phone needs to be properly added.

 

[tcaF0xd]

Ok, on most link2SD setups, the ext2 partition is typically the FIRST partition on the card, with the more conventional FAT32 partition being second. Try that.

Finally, it might be worth it to get in contact with the link2SD developer and explain your problem and findings. It may be that there's a bug in the app he needs to fix, or simply that support for that phone needs to be properly added.

Oh ok, the Link2SD description page says to make the new partition the second partition by reducing the size of the first partition.

I think Link2SD is a victim of the "sh" infinite loop issue because as I mentioned I also get this same infinite loop if I do just about any command from the Terminal app when root. Although, if I just SSH in, I have no problem running commands as root or launching a new shell as root. If only I had pstack or gdb to get a stack trace... Ugh, I don't feel like cross compiling pstack, I can't imagine all the dependencies. I guess I could build a library with statically compiled in dependent libraries?

 

[Joelgp83]

Ugh, I don't feel like cross compiling pstack, I can't imagine all the dependencies. I guess I could build a library with statically compiled in dependent libraries?

Have you tried to install "Busybox" from the Play store yet? It is pretty much an app that tries to install a bunch of compiled-for-android utiilities from linux. Perhaps ptrace is one of them?

Either way, get it. Something in there may help you diagnose the issue.

As far as statically compiliing the dependencies, I don't see why not. You're not planning on using ptrace or the other utilities for a long enough time that you'd have wildly different dependecy verisons on this phone, and storage space probably isn't going to be an issue for the specific utilities you want to use.

 

[tcaF0xd]

Have you tried to install "Busybox" from the Play store yet? It is pretty much an app that tries to install a bunch of compiled-for-android utiilities from linux. Perhaps ptrace is one of them?

Either way, get it. Something in there may help you diagnose the issue.

As far as statically compiliing the dependencies, I don't see why not. You're not planning on using ptrace or the other utilities for a long enough time that you'd have wildly different dependecy verisons on this phone, and storage space probably isn't going to be an issue for the specific utilities you want to use.

Cool, I didn't know about that. I installed it and gave it root permission but it is having problems and just goes into a loop (although I checked "top" and nothing is burning CPU). I see various messages like this in logcat:

D/su (11669): Found file /data/data/com.noshufou.android.su/files/stored/10060-0
D/su (11669): Comparing '/system/bin/sh' to '/system/bin/sh'
D/su (11670): Executing exec /system/bin/am broadcast -a com.noshufou.android.su.RESULT --es socket '' --ei caller_uid 10060 --ei allow 1 --ei version_code 18
D/su (11669): 10060 stericson.busybox executing 0 /system/bin/sh using shell /system/bin/sh : sh
D/su (11682): Found file /data/data/com.noshufou.android.su/files/stored/10060-0
D/su (11682): Comparing '/system/bin/sh' to '/system/bin/sh'
D/su (11683): Executing exec /system/bin/am broadcast -a com.noshufou.android.su.RESULT --es socket '' --ei caller_uid 10060 --ei allow 1 --ei version_code 18
D/su (11682): 10060 stericson.busybox executing 0 /system/bin/sh using shell /system/bin/sh : sh

 

[Joelgp83]

I'm afraid we've reached my current limit of decoding linux jargon/output. I don't see anything in there that is an obvious error, but I do know that on a properly working and rooted phone busybox doesn't have any trouble installing its stuff.

I'd still suggest getting in contact with the developers of both link2SD and busybox and reporting your findings, they may have knowledge of an existing workaround or something.

I agree with you that all signs point to something being borked in the stock ROM, though. While this is kinda the nuclear option, are their any custom ROMs avaible for your phone? If CyanogenMod is available and in a stable state, use that. They tend to be somewhat sane when it comes to stuff like this.

 

[tcaF0xd]

Cool, I didn't know about that. I installed it and gave it root permission but it is having problems and just goes into a loop (although I checked "top" and nothing is burning CPU). I see various messages like this in logcat:

D/su (11669): Found file /data/data/com.noshufou.android.su/files/stored/10060-0
D/su (11669): Comparing '/system/bin/sh' to '/system/bin/sh'
D/su (11670): Executing exec /system/bin/am broadcast -a com.noshufou.android.su.RESULT --es socket '' --ei caller_uid 10060 --ei allow 1 --ei version_code 18
D/su (11669): 10060 stericson.busybox executing 0 /system/bin/sh using shell /system/bin/sh : sh
D/su (11682): Found file /data/data/com.noshufou.android.su/files/stored/10060-0
D/su (11682): Comparing '/system/bin/sh' to '/system/bin/sh'
D/su (11683): Executing exec /system/bin/am broadcast -a com.noshufou.android.su.RESULT --es socket '' --ei caller_uid 10060 --ei allow 1 --ei version_code 18
D/su (11682): 10060 stericson.busybox executing 0 /system/bin/sh using shell /system/bin/sh : sh

Okay, well, I uninstalled that BusyBox app and installed BusyBox Installer, another one, and that works fine. I can see plenty of busybox apps, but unfortunately can't find gdb, pstack, strace, or anything like that....

 

[Joelgp83]

Darn. Looks like compiling your own is the next step, then.

 

[tcaF0xd]

Okay, well, I uninstalled that BusyBox app and installed BusyBox Installer, another one, and that works fine. I can see plenty of busybox apps, but unfortunately can't find gdb, pstack, strace, or anything like that....

Sweet, I found a statically compiled GDB that works on ARM here: gdb for android

Now I just need to attach to the runaway "sh" and hopefully there will be symbols! ...

 

[tcaF0xd]

Sweet, I found a statically compiled GDB that works on ARM here: gdb for android

Now I just need to attach to the runaway "sh" and hopefully there will be symbols! ...

Ugh. It's not figuring out the symbols:

# ./gdb /system/bin/sh...
(gdb) attach 987
Attaching to program: /system/bin/sh, process 987...
(gdb) thread apply all bt
Thread 1 (process 987):
#0 0xafd0c3cc in __wait4 () from /system/lib/libc.so
#1 0xafd18228 in wait3 () from /system/lib/libc.so
#2 0x00010388 in ?? ()
#3 0x00010388 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) disas 0x00010388
No function contains specified address.

I guess the fact that frames #2 and #3 are in 0x10388 does suggest it's in a recursive loop of some sort (or the stack walker is just confused).

Can someone please check on their phone if /system/bin/sh has these same owners/groups/perms?

# ls -l /system/bin/sh
-rwxr-xr-x 1 root shell 82848 Aug 5 2011 /system/bin/sh

Is there a "rooted" sh binary? Remember that I didn't root this phone in the "normal" way (I don't even know what "normal" rooting is).

 

[tcaF0xd]

Ugh. It's not figuring out the symbols:

Well, I rebuilt android-2.3.4_r1 from source, and modified build/core/definitions.mk to build "sh" with symbols, uploaded sh to the phone and re-attached to an sh with 100% CPU and here are a couple of stacks that I caught:

(gdb) bt
#0 0xafd11fe0 in __errno () from /system/lib/libc.so
#1 0xafd24560 in __set_errno () from /system/lib/libc.so
#2 0xafd24576 in __set_syscall_errno () from /system/lib/libc.so
#3 0xafd18228 in wait3 () from /system/lib/libc.so
#4 0x00010388 in waitproc (block=-61456, jp=<value optimized out>, status=<value optimized out>) at system/core/sh/jobs.c:1164
#5 0x000109b2 in dowait (block=1, job=<value optimized out>) at system/core/sh/jobs.c:1055
#6 0x00010b32 in waitforjob (jp=0x1d580) at system/core/sh/jobs.c:992
#7 0x0000c094 in evalcommand (cmd=<value optimized out>, flags=<value optimized out>, backcmd=<value optimized out>) at system/core/sh/eval.c:1072
#8 0x0000b7ea in evaltree (n=0x1c620, flags=0) at system/core/sh/eval.c:297
#9 0x00011994 in cmdloop (top=<value optimized out>) at system/core/sh/main.c:260
#10 0x00011d62 in main (argc=<value optimized out>, argv=<value optimized out>) at system/core/sh/main.c:212
(gdb) continue
Continuing.
^C
Program received signal SIGINT, Interrupt.
0xafd0c3cc in __wait4 () from /system/lib/libc.so
(gdb) bt
#0 0xafd0c3cc in __wait4 () from /system/lib/libc.so
#1 0xafd18228 in wait3 () from /system/lib/libc.so
#2 0x00010388 in waitproc (block=0, jp=<value optimized out>, status=<value optimized out>) at system/core/sh/jobs.c:1164
#3 0x000109b2 in dowait (block=1, job=<value optimized out>) at system/core/sh/jobs.c:1055
#4 0x00010b32 in waitforjob (jp=0x1d580) at system/core/sh/jobs.c:992
#5 0x0000c094 in evalcommand (cmd=<value optimized out>, flags=<value optimized out>, backcmd=<value optimized out>) at system/core/sh/eval.c:1072
#6 0x0000b7ea in evaltree (n=0x1c620, flags=0) at system/core/sh/eval.c:297
#7 0x00011994 in cmdloop (top=<value optimized out>) at system/core/sh/main.c:260
#8 0x00011d62 in main (argc=<value optimized out>, argv=<value optimized out>) at system/core/sh/main.c:212

Looks like it's waiting for some sort of child or job to finish. I see there is a TRACE macro in sh, so maybe I'll turn that on...

 

[tcaF0xd]

Looks like it's waiting for some sort of child or job to finish. I see there is a TRACE macro in sh, so maybe I'll turn that on...

I'm replying just to document things in case anyone else has to go through the same thing.

So, sh has some built-in tracing. One problem is that it defaults to write to ./trace, but the sh command's cwd is /, and there's no filesystem mounted here, so I changed it to /mnt/sdcard/trace in show.c. Then I updated Android.mk to compile in DEBUG mode:

LOCAL_CFLAGS += -DSHELL -DWITH_LINENOISE -g -DDEBUG=2 -fno-inline

Running with this mode, it's clear what's looping. I'm running the "id" command after running "su" in the android terminal and it hangs after it prints the output of id...

pid 27404, evaltree(0x1c650: 1, 0) called
evalcommand(0x1c650, 0) called
evalcommand arg: id
searchexec "id" returns "/system/bin/id"
makejob(0x1c650, 1) returns %1
forkshell(%0, 0x1c650, 0) called
commandtext: ps->cmd 21590, end 21592, left 198
"id"
In parent shell: child = 27427
waitforjob(%1) called
dowait(1) called
Child shell 27427
normal command: "id"
wait returns pid -1, status -1096214220
dowait(1) called
wait returns pid -1, status -1096214220
dowait(1) called
wait returns pid -1, status -1096214220
dowait(1) called
wait returns pid -1, status -1096214220
dowait(1) called

The "wait returns" and dowait calls continue infinitely. Now to figure out what this means!

 

[tcaF0xd]

The "wait returns" and dowait calls continue infinitely. Now to figure out what this means!

Status -1096214220 is WIFSIGNALED with WTERMSIG = 52 but I can't even find signal 52!

 

[Joelgp83]

I forgot, most of the time you need a modded kernel to get the linking to work, at least that's what you gotta do for Cyanogenmod 7.2 on the prevail.

There's a guy here named Koumajutsu, he's done stuff similar to this to get linking to work. He might be able to help point you in the right direction....but I can't speak for him.

 

[tcaF0xd]

I forgot, most of the time you need a modded kernel to get the linking to work, at least that's what you gotta do for Cyanogenmod 7.2 on the prevail.

There's a guy here named Koumajutsu, he's done stuff similar to this to get linking to work. He might be able to help point you in the right direction....but I can't speak for him.

Very interesting, thank you. I certainly don't have a modded kernel, so maybe that's the problem. I will try to message Koumajutsu, but if you can also message him and ask him to look at this thread, there's probably more chance he will read your message. Thanks so much!

 

[tcaF0xd]

I forgot, most of the time you need a modded kernel to get the linking to work, at least that's what you gotta do for Cyanogenmod 7.2 on the prevail.

There's a guy here named Koumajutsu, he's done stuff similar to this to get linking to work. He might be able to help point you in the right direction....but I can't speak for him.

I finally fixed the unending "sh"!! This was the "problem" (I don't really know the root cause):

system/core/sh/jobs.c

while (jp->state == JOBRUNNING) {
dowait(1, jp);
}

I can tell by using ps that each of these shell spawns is finishing, but for some reason the parent is not seeing that the forked child job is finished (getting the signaled error above). In any case, I just broke out of the loop after some iterations and now I can finally use the terminal emulator fine as root without these infinite loops.

However, I'll wait if Koumajutsu can comment on the broader symbolic link issue before I try Link2SD or anything else...

 

[Koumajutsu]

With the replacement of Dalvik-cache that I've done for the Prevail (i.e. SSM) I actually mount an sd-ext partition to /data/dalvik-cache. As you've found, symbolic links seem to fail. This is the result of what's referred to as a 'race condition', basically meaning one process is trying to read the file while the location the file is pointed to is not available yet (read: /cache hasn't mounted yet)
What I did for the Prevail is replace lines in the init.rc on the ramdisk, (on the prevail it's bundled into the kernel, I'm not sure how Kyocera does it's base filesystem) initially adding a mount line right after the mount for /data to use a partition of the sd-card instead as /data/dalvik-cache.
if the partition wasn't there, it wasn't a big deal, the phone took a couple extra seconds to boot while it either mounted the additional partition, or timed out and moved on.

I'm not sure how helpful this information is for you because it sounds like you have everything you need in place for Link2SD to work. give it a try, worst case, you have to roll back to a backup.

of course, ALWAYS make a backup

 

[tcaF0xd]

I'm not sure how helpful this information is for you because it sounds like you have everything you need in place for Link2SD to work. give it a try, worst case, you have to roll back to a backup.

Darn, I have a second primary fat partition, but Link2SD complains "Mount script cannot be created" without any other detailed error messages. I don't see any error messages in logcat. I guess I will have to do your approach, thanks for providing details! One question: you said you mount sd-card right after /data - so you just mount on top of the already mounted /data partition? Can you please list the full mount commands you use in init.rc? Thanks!

I'm also not sure how to modify init.rc in the the ramdisk. If you have any tips or pointers, that would be great!

 

[divinebovine]

Bump. This is relevant to my interests. I too would like more details, Koumajutsu. I've considered just mounting my entire /data as a partition on SD and I didn't realize there'd be a race condition where the SD card isn't available when it's time to mount.

 

[tcaF0xd]

Bump. This is relevant to my interests. I too would like more details, Koumajutsu. I've considered just mounting my entire /data as a partition on SD and I didn't realize there'd be a race condition where the SD card isn't available when it's time to mount.

As part of my discussion with the developers of break_setresuid that allowed me to root my Kyocera, he mentioned this:

"You have to dump partition that it starts with "ANDROID!" and size is about 10MB.
Edit system("/system/bin/sh") to run dump command such as dd in main.c ."

So I guess that's step #1, so then that presumably gives the compressed kernel image, but I'm not sure if that's where the filesystem is stored, or how to modify the files on such a filesystem.

 

[Koumajutsu]

Sorry it took so long to get back to you. I Kinda forgot to check back

Darn, I have a second primary fat partition, but Link2SD complains "Mount script cannot be created" without any other detailed error messages. I don't see any error messages in logcat. I guess I will have to do your approach, thanks for providing details! One question: you said you mount sd-card right after /data - so you just mount on top of the already mounted /data partition? Can you please list the full mount commands you use in init.rc? Thanks!

I'm also not sure how to modify init.rc in the the ramdisk. If you have any tips or pointers, that would be great!

Here is what the traditional Prevail SSM mount strings look like:
/dev/block/stl13 is the internal flash memory (what is normally /data)
/dev/block/mmcblk0 is the prevail's SDcard.
This configuration makes use of a card that's been partitioned into 3 parts, the FAT part, and 2 EXT4 partitions

mount ext4 /dev/block/stl13 /data rw noatime nosuid nodev barrier=0 data=writeback nobh nodiratime noauto_da_alloc
mount ext4 /dev/block/mmcblk0p2 /data/app rw noatime nosuid nodev barrier=0 data=writeback nobh nodiratime noauto_da_alloc
mount ext4 /dev/block/mmcblk0p3 /data/dalvik-cache rw noatime nosuid nodev barrier=0 data=writeback nobh nodiratime noauto_da_alloc

Bump. This is relevant to my interests. I too would like more details, Koumajutsu. I've considered just mounting my entire /data as a partition on SD and I didn't realize there'd be a race condition where the SD card isn't available when it's time to mount.

yeah, if you want to go with mounting the SD card as the /data partition, you have to make sure you don't mount the internal flash first. Otherwise things like WiFi will stop working when their config files get created on the internal, then get hidden when the SDcard mounts over them. Try it! it's kinda interesting to see the effects and was a great learning experience for me into how the init.rc scripts do their thing

 

[tcaF0xd]

was a great learning experience for me into how the init.rc scripts do their thing

Hey Koumajutsu, thanks for the reply. Do you have any thoughts or a walkthrough/tutorial on how to actually update the init.rc scripts? I presume we have to somehow extract something from the phone (using dd?), then update the init.rc scripts in this blob file and then put it back on the phone. Is this what people talk about when they say they "flash" their phone? Any tips?

 

[divinebovine]

This would not be related to flashing, nor would you use dd. Those are very different, replacing entire operating systems and file systems. We're just talking about modifying individual files.

Underneath the shiny exterior we see on our phones' screens, Android is made of linux and linux-like guts. In linux, an early and integral part of the boot process is to use a program called init which reads a script (or a number of scripts). In Android, the most important one is /init.rc.

You could edit it in a variety of ways. You could use a root explorer on the phone (like ES File Explorer) with a text editor, or you could edit it from the adb shell prompt (most likely using vi), or you could transfer the file to your computer (using adb pull or your favorite way of moving files between root-only areas of the phone and your computer) to edit it there. Be sure to back up the file between edits, and be sure to do a nandroid backup before you start too.

You will be most comfortable and effective if you have some serious linux experience.