Proposal/Theory for External SD storage solution on F6Tips


  1. JVene

    JVene Well-Known Member

    In know the audience will range from the inexperienced to experts, and while this discussion is intended to include all, be warned:

    Nothing here is a downloadable solution for novices or casual users, but is a discussion for those of us who know how to unbrick our phones when we experiment with dangerous ideas, understand the Linux command line, the development tools (specifically adb), and know how to partition an SD card through ADB from the linux command line served from the phone without some downloaded app.

    If that's not you, you may still find this discussion interesting (and hopeful regarding the storage problem), but only try any of this if you know what you're doing.

    Read this like the notes from an engineer's desk after running some experiments.

    What I'm posting here is a working theory tested in my F6 (and genymotion) for a grand solution to the typical storage problem. We're all running out of room when there's an empty 32 GByte SD card installed. We all know that the various "link" and swap solutions aren't really doing the trick. You may have even gotten your phone's storage page to claim there's 29Gbytes of free space, but realize after trying to use it that it's not entirely true.

    Well...this may do the trick, but it's not without a compromise, but one chosen to be outside the area of compromising storage.

    If we could edit init.rc we could simply alter the boot sequence so the main directory of interest, /data, would no longer mount from mmcblk0p15 (maybe it's 14, I'm not looking at it right now)..but from some other source, like the external SD card (typically mmcblk1p1). mmcblk0 is the base node of the 4 Gbyte internal SD card, and mmcblk0p15 is a 1.2 Gbyte partition on that card, holding the content of the /data directory.

    Substituting mmcblk0p15 for mmcblk1p1 (or p2, if the external card has two partitions) would provide a genuine primary source from the SD card for /data, providing THE solution to the storage problem.

    However, the F6's boot loader is locked, so every time the phone boots, init.rc, much of the root directory structure, and anything we might custom configure is restored to "default" conditions.

    If not for that lock, this could be as simple as doing something like this just one time:

    cp -rp /data /storage/ext2filesys/data

    which duplicates /data and all it's contents (pretty much all the applications and their data), including permissions and links.

    and then during boot:

    mount -o rw,bind /storage/ext2filesys/data /data

    Assuming that ext2filesys represents a partition from the external SD card formatted in ext2 filesystem. It would be more like:

    mount -o rw -t ext2 /dev/block/mmcblk1p2

    If we were making a custom ROM and installation on the external card for this sort of thing, but we're not. We're using the bind option to mount a directory over another directory instead, but it's similar.

    ....and, of course, a few other machinations in detail.

    However, the fact that we can't edit init.rc (or it's cousins) without cracking the boot loader lock means that we can't perform this mount before zygote is launched. Zygote is name of the process running the Dalvik runtime, it's basically what everyone see's as Android, running as a process inside Linux.

    Most here are familiar with smanager and/or universal init.d, or something similar. They run scripts "during boot"....but, they run them AFTER zygote is running, because they're Android applications, not Linux scripts. At that point zygote (another name for Android) is already dependent on the content of /data, and in particular some of it's major and minor subdirectories. Google's playstore is key among them, because it's a wimpy, whiny app that will repeatedly complain once /data is swapped out underneath it, making the phone all but unusable.

    So....the basic novel idea I'm proposing is to kill the zygote.

    At this point the astute readers probably have an expression of utter disgust.

    It's a FUGLY proposition, but one that happens to work so far as I can see.

    The idea is to view Android in two parts. The first part is Linux. It boots up and operates like a stripped down but otherwise standard Linux build. Once it's going, which happens rather quickly, it launches zygote. That's when you see the boot logo.

    At that point it's already too late to REALLY substitute storage, unless you're willing to view zygote (and thus the Android PORTION of this two piece structure) as individually rebootable - by killing zygote.

    Init, the grandparent process of the entire Linux implementation here, will restart zygote, probably within 1ms. When it does, it happens AFTER the substitution mount command example above, making /data serve from the external SD card.

    You can try this without script editing yourself, if you're a bold experimenter. Simply arrange for a duplicate of /data on an ext2 filesystem, mount it (driving Google's apps nuts), then ps | grep zygote, note the pid and then kill zygote. It LOOKS like the phone reboots, but it's only the zygote rebooting (and thus Android) - NOT the entire Linux operating system underneath Android.

    And zygote will boot as if for the first time, after having reconfigured Linux the way we want.

    So far I'm observing zero complications and 100% satisfaction on the storage problem with this approach.

    No app by app fiddling with links, or move to phone - just space. Actual, real space.

    If you're familiar with fdisk, mkfs.ext2, mkfs.vfat (for a fat32 partition of you want one), the mount command and such, you can probably put the pieces together yourself.

    Now, making a script that can grep the zygote process from the output of ps, then sed and/or awk that to get the pid to fashion a kill command is an ugly looking bit of code which causes me to reach back 30+ years into my early Unix work, which I've happily let slip from memory, though Google provides.

    Frankly I'm so much more comfortable with the NDK, and the stand alone toolchains that I'm motivated to write a C/C++ command line utility to do it instead, but maybe I'll try my hand at the script for a bit.

    The basic plan is this:

    Upon power up, Android (as zygote) initializes from the default, internal version of /data as mounted from mmcblk0p15 - the stock version of /data, after rooting, equipped with smanager or universal init.d to initiate a script in /system/etc/init.d.

    That goes through machinations to mount the ext2 partition (they don't automatically mount easily on the F6) - then, calls another script on the ext2 partition to continue. I suggest this so that simply pulling the SD card boots a "standard" configuration based on the stock version of /data.

    The second script, probably launched as a new thread (using the & at the end of the command, for example), would perform the substitution for /data served out of mmcblk1p1 or p2, then kill zygote to reboot it.

    An interesting consequence is that, from the Android perspective, you have two operating systems. One on the internal card at mmcblk0p15, and the other on the external card at mmcblk1p2 (or p1 if you choose a single ext2 partition).

    This means that all applications you install after such a modification will end up in the external partition. If you pull the card and reboot, with the design plan mentioned above, the "stock" or internal version of /data remains...without those applications installed.

    Or any photos you've taken, or downloads...etc.

    This essentially abandons that admittedly puny 1.2Gbyte partition that came with phone, relegating it to a boot and self destroy function, simply to launch the version from the external card.

    Until someone cracks the boot loader lock (before the phone is outdated and replaced, that is), this may be the only way to gain REAL open storage on the external SD card.

    Anyone up for a ride?

    Advertisement
  2. JVene

    JVene Well-Known Member

    On second thought....

    That really is a FUGLY proposed theory.

    And, it does have pitfalls - under certain conditions rebooting zygote that way doesn't quite do so well.

    I just found another way I'm going to tinker with today.

    I didn't realize that the entire /system subdirectory survives reboots.

    That's key.

    Here's why.

    As I pointed out above, what we don't have, and would prefer, is a way to run script code before zygote launches. Changes to init.rc are out.

    But, taking a cue from busybox and toolbox, and newly recognizing that changes made to content in /system/bin survives reboots, I realize now that at a key point in init.rc, they call mount.

    mount is in /system/bin, with another link in /system/xbin (when busybox is installed).

    They way toolbox and busybox work is that they read the command line and perform functions as aliases to real commands, like this:

    busybox chmod a+rw filename

    The real Linux command is

    chmod a+rw filename

    Now, for toolbox, chmod is a link to toolbox, and in /system/xbin, when busybox is installed, chmod is a link to busybox.

    We may type

    chmod 777 filename

    But chmod links to toolbox, which calls toolbox instead of some chmod code.

    On standard Linux, chmod is a c program. On Android implementations it's not. It's a link to toolbox, which then reads the command which launched it. In the arguments it receives it can tell the command line reached the toolbox executable via chmod.

    This is opposed, to say, the same thing happening to reach toolbox (the exact same executable) via mount, or some other command.

    Here's what I've been looking for.

    I can substitute the mount command. I can change the link from toolbox to, say, customtoolbox. I intend to make customtoolbox.

    It will be a simple c program, compiled with the standalone toolchain under NDK9b, probably clang3.3, targeting ARM and x86 for Android's implementation of Linux.

    This is the way something like busybox is written.

    The objective will be, in default behavior, to forward the command to toolbox so operating is exactly as before.

    However, I should be able to determine, within customtoolbox, if the OS is booting. I've not yet worked out the details on that point, dozens come to mind immediately and I'll have to search for the best fit.

    The point is that when init.rc calls mount, it will be before zygote launches. Factually, it may be before much of anything is mounted - it should end up calling mount very early on in the boot process (and several times subsequently).

    Any command could be intercepted in this way....cp, ls, mount....just about any of them.

    At the time customtoolbox is about to forward a command to toolbox, it can check to see if this is the first occasion that's happened since boot, and to subsequently check to see if sufficient storage resources have already been mount - such that customtoolbox can call scripts in, say, /system/etc/custominit, in a style similar to /system/etc/init.d, but long before zygote is involved.

    This would be a great moment to look for /data, unmount it (if required), and subsequently mound a new version of content for /data from the external SD card.

    I'm going to work on it later tonight.

    Anyone else interested in experimenting too?
  3. JVene

    JVene Well-Known Member

    Big shout out to Trevor Philips , and thanks.

    I've had a Eureka moment on the theme above.

    /system/bin can indeed retain modifications made to content, and there's an even better approach to substitution that just hit me like a ton of bricks.

    The app_process binary.

    When init.rc executes, there comes a point when the service zygote is to be started. The code from init.rc on the F6 is:


    service zygote /system/bin/app_process -Xzygote /system/bin --zygote --start-system-server
    class main
    socket zygote stream 660 root system
    onrestart write /sys/android_power/request_state wake
    onrestart write /sys/power/state on
    onrestart restart media
    onrestart restart netd
    onrestart restart surfaceflinger

    What's key here is /system/bin/app_process.

    The source code for app_process (like all the source of Android) is online.

    I can rebuild the app_process binary with a slightly modified custom version, which will survive reboots.

    It is ideal given the circumstances (of a locked bootloader).

    I'm triple checking I have the same version of source as that of the 4.1.2 on the F6.

    The code is extremely simple. app_process is used to launch various android applications, and intercepting here is perfect:

    Code:
        if (zygote) {
            runtime.start("com.android.internal.os.ZygoteInit",
                    startSystemServer ? "start-system-server" : "");
        } else if (className) {
            // Remainder of args get passed to startup class main()
            runtime.mClassName = className;
            runtime.mArgC = argc - i;
            runtime.mArgV = argv + i;
            runtime.start("com.android.internal.os.RuntimeInit",
                    application ? "application" : "tool");
        } else {
            fprintf(stderr, "Error: no class name or --zygote supplied.\n");
            app_usage();
            LOG_ALWAYS_FATAL("app_process: no class name or --zygote supplied.");
            return 10;
        }
    
    

    Where the "if (zygote)" code initializes the main instance of the zygote engine, and the "else if (className)" block launches an app in an existing zygote process.

    This moment is an excellent opportunity to synchronize things. At the zygote initialization I can block while running scripts, certain that zygote is not yet launched - then let it continue as if nothing was interrupted.

    I'm going to prepare for editing / rebuilding this on the F6 tonight.
  4. JVene

    JVene Well-Known Member

    Update:

    The download of the source for Android OS 4.1.2 is a little over 8 Gbytes, which will take a while. That means my experiment will likely not commence until late Saturday. I could attempt just to compile the app_process binary by itself, but it is dependent on OS shared libraries targeted at 4.1.2, and I'd prefer to install the build into an AVD and experiment with it as a proxy for the F6, so I'm loading the whole thing. Building it will take some time, too.

    In the meantime, while studying the potential hinted in my previous post, I have some observations to add.

    This can do more than merely remount /data on the external SD card. It can also customize the OS considerably.

    As you probably realize, when Linux boots the stock software is restored, destroying any interesting customizations one might hope for in a custom ROM. Until there's a sure way of unlocking the boot loader, customization could still be realizable if we accept a few inconveniences in the mix.

    First, this is still going to be 4.1.2. That won't change. This technique can't easily substitute a new kernel. However, 4.1.2 could be considerably customized, simply because app_process will have control in Linux before zygote is launched, and will block the initiation of zygote until all adjustments are completed.

    This means, for example, there could be a custom boot logo sitting on the SD card, waiting to be copied into position. The original is obviously going to be in position at first during boot (every time), but once it's placed, just prior to zygote's initiation, the new app_process could substitute a replacement. The zygote process is what shows the boot logo. You see nothing on the display until zygote shows it to you during a normal boot.

    It should be possible to ensure that those pesky, bloated applications we can never delete are removed, too - and it should be possible to replace some of them with better ones.

    Put another way, outside the realm of updating the kernel, a great many customizations should be possible simply because the boot cycle has a new point of custom control, via a new version of app_process.

    My first experiments will be in an AVD, hopefully during the weekend. When I think it's ready, I'll post it with usage hints (no, not instructions - if you're not already familiar with the adb shell, root, commands like mount, cp, mv, fdisk, mkfs.ext2 and others, this isn't ready for you). Experience users would then be able to experiment until I can fashion something more suitable for a wider audience.
  5. lucasmun09

    lucasmun09 Well-Known Member

    Nice job and really nice documentations you got there! But I advise you to pause what you are doing for a moment... Optimus F6 may be receiving the 4.4 kitkat from the lg soon. LG to skip Android 4.3 and update many of its devices to Android 4.4

    What you are doing is really great but I am afraid that certain things that you are trying to do may not be compatible with the new KitKat's structure.... Not that I'm going against what you are doing lol.... but I can't just see someone who is working very hard to solve a problem ending up becoming obsolete because of the incompatibility...

    morgankslade likes this.
  6. JVene

    JVene Well-Known Member


    Thanks for the heads up (I didn't know they were planning 4.4 soon) - and let me address the point and how this relates to the Xpose framework.

    I'm new here, so a touch more self introduction may be in order. I have over 30 years of experience as a software engineer, with targets ranging from business applications to device drivers, with stopovers in video post production products, 3D graphics engines, robotic manufacturing and machine control...a great deal of the entire map. Unix has been at the heart of my work (Linux, too), but of course Windows and Mac have taken my attention at key points over the years. My point is, this is so far from my 10th rodeo that it's familiar territory. Particularly regarding your point of the platform changing out from underneath my work.

    Also, it's not really hard work here. I have to share the Internet with a family of users, so occasionally my huge downloads cause family .... uh...discussion ;) For that, I accept the occasional delay. I'm simultaneously dealing with everything from homework help to laundry to a flat tire to 3 simultaneous development projects (somewhat related to each other) targeting iOS, Android, Tizen and Blackberry - all temporarily interrupted by a frantic call by an OLD friend (colleagues about 10 years ago) sweating bullets because he accidentally deleted an entire corporate database on AIX based RS-6000 servers at his job, at about 2AM - for which there's no "undelete" feature - so I had to reconstruct the directories for him before business opened at 6am.

    The project discussed here is just a public sharing of work I'm doing for my son's phone, a Christmas gift. He's 12 - wants a few games in the 300 to 900 Mbyte download range, and I don't want to have to hear about it ongoing :). I assume the result will be of interest, because my own experiments thus far demonstrate it's a complete solution (caveat on your very valid point included, because that too affects my son's phone).

    I found that this approach I'm settling on happens to be related to the Xpose framework. Xpose also modifies app_process as the primary hook for performing it's feature set. The purpose is entirely different, but the key point of contact is the same.

    I pointed this out to Xpose primary developer, rovo89 on xda-developers, because conceptually there are only about 2 or 3 lines of code to be added to app_process in order to provide the needed feature, and since BOTH ideas replace app_process, they would be mutually exclusive without either some cooperation on the point, or a fork of the Xpose source only to provide an alternative app_process binary.

    To that end, I've taken careful review of the 4.4 Android source. For app_process there's only one source file, app_main. It's really tiny, extremely simple. This code hasn't changed significantly since about 1.6 or earlier. There are minor differences between froyo and ICS (enough that versions of app_process must be built for either), and I have interest in producing a froyo version as well.

    There's no difference between that source in ICS, JB...basically 4.0 through the current master branch of 4.4+.

    The code itself produce a tiny binary, which does most of it's work by dynamically loading a shared library, to inherit the runtime object of the VM (that way, it adapts to the local version of Android nicely - app_process built for 4.04 works fine in 4.1.2, for example).

    Another build may be required for 4.3 and yet another version for 4.4, primarily due to differences in the shared library it references - but so far I've found that in the emulators it makes very little difference. Providing 3 or 4 versions of app_process is a simple matter. It takes less than 4 seconds to build app_process - less if my machine weren't also busy transcoding video in the background (long story).

    Key to the design plan is the fact that new code in app_process doesn't have to do much. It simply runs a script - a hook into making everything else happen. If the script isn't there, the code just continues doing what it's always done - launching zygote. If the script is there, that's where all the work required is done - not in app_process.

    When an update occurs OTA or by recovery, the /system/bin directory would undoubtedly be updated, and the stock app_process replaced. The planned approach to the storage solution would simply boot the phone in a "stock " arrangement when this happens. A simple subsequent replacement of app_process targeting 4.4 would restore the feature completely, with one caveat. The /data directory on the SD card would likely be the older version (the OTA update from the service provider would probably have ignored it, and certainly a recovery flash would behave as though it were not there). This means a simple script which compares the content of the /data form the internal card to the /data version on the external card would suffice to align the external correctly (or the reverse if an OTA update happens to route to the external SD card's version of the content).

    Like I posted early on, this is a thread documenting a little research and experiment on an F6 in my possession, and soon I'll post an app_process build (or 3) which can be used by knowledgeable members here who may wish to experiment, fully prepared to recover their devices.

    After a little more effort on the project it might be more suited for lesser experienced individuals, or for someone else to take the concept further, along with work done with the Xpose framework itself.
  7. JVene

    JVene Well-Known Member

    Oh, nearly forgot to add this interesting possibility:

    During this critical moment in the boot cycle, with zygote not yet initialized, it may be possible to substitute the Dalvik VM. This is one reason I'm loading the entire source for several versions of Android. My hope is that, if desired, a build of the Dalvik runtime generated with the Linaro toolchain could be substituted. If the original ROM was not built with Linaro, this could offer a considerable performance boost. I'll get to that tinkering after storage is solved, but like the Xpose framework, this hook (app_process) offers a wide range of possible enhancements to devices with locked bootloaders typically associated with custom ROMs.
    morgankslade likes this.
  8. lucasmun09

    lucasmun09 Well-Known Member

    Just wondering with all the respect, have you tried this? http://androidforums.com/lg-optimus-f3/801368-mod-app2external_sd-fixed-duplicate-files.html
    I see that your problem is with your son having a small amount of storage and is not able to install apps larger than 600mb and so on. This swaps the sd and internal partitions and enables the user to download these such apps. It is working fine for me. If I missed something I am very sorry but that is what I have taken in from the post above. :D

    Also, your goal is to swap the partitions not just the data files but the actual mounting point I presume so when I install the app, it installs the app installment and the data directly to the sd card without emulating it to be one.

    morgankslade likes this.
  9. JVene

    JVene Well-Known Member

    Yes, I tried several approaches along the same basic lines.

    If you examine the script for app2external_SD, you'll notice it mounts /storage/sdcard0 over a directory created /data/internal_sd. If you examine the stock device, you'll realize /storage/sdcard0 doesn't actually represent a card, or a partition of a card, it represents a mount of the directory /data/media. It's just a directory inside /data. It's also where many of the downloads are directed.

    /data, however, is a partition of the internal card (it's mounted from an entry in /dev/block, not another directory in a filesystem previously mounted). The /storage/sdcard0 intent of the stock phone is simply to put stuff in /data/media, alongside everything in /data/app, /data/data and other material under /data. /data is the mount of a 1.2Gbyte partition on the internal SD card. What is being swapped is NOT the internal_SD card, but a representation of a subdirectory masquerading as an internal sd card.

    Most downloads route to a location in /data/media via /storage/sdcard0 or some other approach. However, data downloaded by other applications may end up in some other directory underneath /data/data, which app2external_SD hasn't re-routed, and is sharing that 1.2Gbyte storage space.

    In other words, while the internal storage size now appears to be large (and therefore DOWNLOADS go to a larger area), the location for actual applications and their data is still limited - it's still in the 1.2 Gbyte region. You can download a large file, but if the phone is stuffed (like my son's gets) with applications already, that's not going to solve the larger problem - the storage of the application and it's data after it's downloaded.

    Some have reported combining app2external_SD with link2sd. That application works with a different approach. Instead of remapping drives, it does a grander version of what "move to phone" does in the stock usage.

    Move to phone creates a virtual filesystem, mounts it, then points Android to that location for the APK which launches the application (removing it from /data/app, freeing space). Unfortunately, in most cases, it doesn't move material in /data/data. The virtual filesystems created are fixed sizes, created to match the storage requirements of the APK and it's cousins. It won't expand over time, and since application data DOES expand, move to phone doesn't help all that much for material in /data/data.

    Not only does "move to phone" leave /data/data material in that 1.2Gbyte partition, it also doesn't relocate the dalvik-cache directory under /data.

    link2sd helps move more of the material out of the 1.2Gbyte card than move to phone can do, using symbolic links.

    Unfortunately, this still means we're functioning out of a 1.2Gbyte partition of a 4Gbyte sdcard for everything else.

    Put another way, the page in settings which shows storage is wide open can lie to you. It's easily fooled because it doesn't represent the entire map of storage as does the df command in Linux, which you can see through a terminal app or the shell.

    It does help, at first, but you'll run into yet another wall while plenty of room is still open on the external card but none left in that 1.2Gbyte partition on the internal card.

    Most of the other solutions are done application by application (tedious) - and while app2external_SD helps, it too will yield to the reality that only PART of the storage has been remapped.

    At one point I remounted /data, the root of all that. It drove Android nuts until I executed a soft reboot (killing zygote). That works, and soft reboots are well known on many devices, but on the F6 soft reboots are spuriously prone to errors. This phone doesn't really like soft boots. Plus, a soft boot shows the boot logo - and in my first attempt at a solution, that meant two boot cycles (or, when it went wrong, a boot loop lock).

    The engineer in me wanted to open up the phone, desolder the internal SD chip, and put a 32Gbyte chip in it's place.

    Both my son and my wife had objections, so I didn't ;)

    I'm determined, therefore, to do exactly the same thing without the hot iron!

    Grab a free terminal app if you don't have ADB and get to the command line, the type df.

    You'll get a report something like this (though this one is mine, mid experiment)

    Code:
    Filesystem                 Size         Used         Free    Blksize
    /dev                    402.14M       64.00K      402.08M       4096
    /system                   1.19G        1.06G      132.54M       4096
    /mnt/asec               402.14M        0.00K      402.14M       4096
    /mnt/obb                402.14M        0.00K      402.14M       4096
    /data                    29.04G        5.63G       23.41G       4096
    /sns                      7.91M        4.11M        3.80M       4096
    /mpt                     31.54M       12.50M       19.03M       4096
    /persist-lg               7.91M        4.14M        3.77M       4096
    /cache                  788.39M       14.38M      774.01M       4096
    /persist                  7.91M        4.10M        3.81M       4096
    /firmware                63.95M       41.80M       22.16M      16384
    /storage/external_SD     29.04G        5.63G       23.41G       4096
    
    This tells you the real story of storage.

    Notice there's no /storage/sdcard0 ?

    That's because it's not really a device.

    Did you notice /data above? You can df /data to see. Compare it to yours.

    Now try this, the df command directed to interrogate a path:

    df /storage/sdcard0

    Code:
    
    Filesystem                 Size         Used         Free    Blksize
    /storage/sdcard0         29.04G        5.63G       23.41G       4096
    
    
    Yours MIGHT look like mine, because they're routed to the external card. The stock phone would show only a 1.27Gbyte partition here.

    In my device, under experiment, /data is not mounted from /dev/block/mmcblk0p15, as it is in the stock device. It's mounted from /dev/block/mmcblk1p1 (soon to be changed to p2) - the external card.

    Now compare a little deeper:

    Examine with these commands in a shell

    df /storage/sdcard0
    df /data/app
    df /data/data
    df /data/dalvik-cache
    df /data/media

    What space is it showing you? The space of the external card for all these directories?

    On a stock phone, /data/app would look like this:

    Code:
    /data/app                 1.27G      584.55M      716.93M       4096
    
    Representing a 1.27 Gbyte partition, with 584 Mbytes consumed, 717 Mbytes available.

    On my device all of these directories show the external storage space, just like /storage/external_SD on my device. Real, genuine open space.

    Code:
    /data/app                29.04G        5.63G       23.41G       4096
    
    Here it shows I have room for another 23Gbytes of apps.

    If any of yours do not show the external storage space, then the smaller ones are on the internal card and are going to fill up at some point, with plenty of room open on the external card and no room to install applications left.

    The Android storage app can lie to us when tools remount storage partially.
    aaanadie and morgankslade like this.
  10. morgankslade

    morgankslade Well-Known Member

    I am really glad you guys are putting your heads together to help F6 thread out.
    I don't know how you all are doing this, hats off to you.

    Will all this work if we get rumored kitkat update?
  11. JVene

    JVene Well-Known Member


    At the risk of invoking Senator Palpatine's voicever likeness:

    I will make it work, because I have the F6.

    ;)
    SUPPORTLOCAL and morgankslade like this.
  12. morgankslade

    morgankslade Well-Known Member

    In Fat Albert's voiceover likeness:

    Hey, Hey, Hey.
  13. pressy4pie

    pressy4pie Well-Known Member

    very interesting, but over at the metro side of the woods we have loki ( not sure if tmo does or not) and im making a boot.img for stock roms, with said data tweak in the init.rc. anyone in tmo-land wanting to test 4 me?
  14. lucasmun09

    lucasmun09 Well-Known Member

    I would be glad to but im not exactly sure what you are trying to do....
  15. JVene

    JVene Well-Known Member

    loki is an exploit that creates a new boot image bypassing the security method used to lock the ROM, so custom ROM's can be installed.

    It was originally written for some Samsung devices, but two examples of the LG Optimus F6 have appeared.

    The problem, however, as I see it, is that there's a real possibility (and a fairly high one) that it will seriously brick the phone in a way not familiar to even many experienced users.

    Loki happily recognized my "h" metro ROM as an "e" version - and verified that it created a new, unlocked version I could modify.

    Unfortunately the resulting image wasn't valid. The source reveals there are tests used which are too simple, such that it's easy for it to misidentify what it's editing.

    If the match is exact, loki has worked. If not, you can very easily wreak havoc. If you flash backwards to "e" it should work, but that would also be an issue if one wanted to avail themselves of the 4.4 update pending. It would be a wait to get a copy of the 4.4 which could be fashioned by loki to work.

    I like the potential, but I'll personally wait on implementing loki on the F6.
  16. JVene

    JVene Well-Known Member

    Ok, here's something you can try if you qualify yourself as reasonably experienced on tinkering with Android devices.

    If you're completely satisfied with solutions like app2external_sd and/or link2sd, or similar applications, you should probably keep using them. If you discover, however, like I did, that they're not complete solutions and instead want the primary storage of the entire /data directory to mount from the external SD card, this may be what you're looking for.

    As of this post, this is an ALPHA version, not packaged for end users. If you're experience with Android, are familiar with ADB, have rooted the phone, familiar with Linux command line usage and are completely prepared to restore the phone with either LG's tools or a kdz of the stock operating system as a precaution for your own experiments, then you're qualified to try this. If you're not entirely familiar with these concepts then you should wait for this project to advance, but you may find it interesting to follow.

    Read what follows as if you found the notes from an engineers desk detailing an experiment. This explains how it works and how to try it.

    NOTE: I said TRY IT. Some are frustrated with the LG devices, F6 and F3 among them, because of the storage problem, and are eager to try solutions. The link(s) at the bottom are EXPERIMENTAL, and the reason they're posted is for experienced users to assist in PROVING them to be viable. If you're more of an end user looking for a quick solution you can install, this is not yet ready for that.

    If you are knowledgeable and confident to experiment, then consider this a sincere request to try it and post back with problems, observations and suggestions. The intent is to provide something more suitable for the typical user capable of rooting their phone, but little else.

    I have this working on my son's F6 - it can work on yours, or your F3, L9 or most any device with Android 4.0.3 through about 4.2.2, with a really simple and quick move all the way to 4.4.2.

    I type fast and, like Pascal once noted, I don't have time to write a short post, so I wrote a long one.


    This project was started around Jan 1, 2014, prompted by a Christmas gift of an LG Optimus F6 for my son. While the phone is wonderful, the /data directory is mounted from a 1.27 GByte partition on the internal SD card, which easily fills up (that took all of 2 hours to discover). I tried various solutions, from app2external_sd combined with link2sd, as well as manual experimentation using a shell through adb on the phone. Nothing really works. They help, there's no doubt. You can manage to relocate applications to the point where you free up another 300 to 500 Mbytes, maybe more, but that 1.27 Gbyte partition, mmcblk0p15, still fills up eventually. My son is 12. He's already fetched a 600Mbyte GTA 3 and a 900 MByte GTA San Andreas. It's possible to finagle these into the phone, but that's only a temporary solution. Between my son and my wife there will be over two hundred apps installed, I see it coming. /data/dalvik-cache may need 1 Gbyte just for the dex files.

    Well, I'm a software engineer of 30+ years experience, with a long history of Unix and Linux work, so naturally I decided to cobble together a serious solution.

    While there is information that "loki" is available to unlock the boot loader of the F6, there are valid reasons for concern using it. Loki exploits a minor flaw in the security method of the boot cycle, allowing a modified image to be booted instead of the approved factory image. The problem with using it, as I discovered, is that at this point support for the F6 is limited to specific carriers and versions of the OS. Loki happily generated an invalid image for my F6 without warning, and the potential for serious bricking of the phone is fairly high. Rumor has it that Android 4.4 will be coming to the F6, and if you rely on loki to unlock the boot loader as a means of solving the storage problem, you may be out of luck for a long while waiting for loki to support version 4.4 on your combination of carrier and version. I support what loki is doing, but I can't see myself using it for this problem.

    The solution presented here will still work on the 4.4.x version, without waiting. The technical modification is actually simple. It does rely on the fact that LG leaves the content of /system intact between boots. I suspect they do this because they link pre-installed applications from /data/app to /system subdirectories, and want to be able to update them without updating the system.img stored in ROM. Factory resets, as a result, will restore the content of /system, and therefore remove the modifications proposed here. They are, however, easily re-installed.

    I have this working on my son's F6, and the experiments have been repeated on emulators and other devices, including a Kindle Fire first generation on Android 2.3.3 (but I'm not bothering with posts on supporting earlier than 4.0.3 here). It works in all of those situations. Your mileage may vary.

    The goal is to provide a source for the /data directory from the external sd card - all of /data, not just links or subdirectory mounts. We are prevented from doing this because the boot loader is locked, and therefore all changes to the important file init.rc and it's cousins will evaporate at each boot. This would be trivial to solve if we could edit the ramdisk.img, or files in other images, but we can't (reliably, IMO).

    /data is mounted from a partition of 1.27 Gbytes, /dev/mmcblk0p15 (on the F6). The content of other important directories like /data/dalvik-cache, /data/data (and all it's children) and /data/media are all served from that 1.27 Gbyte partition. It isn't practical to mount other sources over these directories because in the stock configuration (which we're bound to by a locked boot loader) we have no way of performing those mounts until after the Dalvik VM has launched, and critical Android applications are already dependent on the content of these directories.

    Early on I tried to mount /data from the external card, which irritates Google's Gapps and it's cousins to the point of repeated pop-up complaints, making the phone unusuable until they're disabled. A soft reboot fixes this, but the F6 is very touchy about soft reboots for some reason. I don't like a soft boot approach either - it's very messy, leaving material in various databases unkempt, sometimes confusing applications.

    What is required is a way to fire a shell script before the Dalvik VM launches. Smanager, Universal init.d and similar tools can't do that. They begin execution from within the Dalvik VM, after it's already running.

    Outside of altering init.rc, the solution is to replace a single small binary file called app_process found in /system/bin. Links at the bottom for alternate version(s) of this binary were built from AOSP source version 4.1.2_r1, which are indentical from 4.0.3 through the current master branch, 4.4.2 - though I'm going to append one built from the 4.4.2_r1 AOSP source to be certain we're set for the 4.4 update. This modified version of app_process blocks the initializatin of zygote, the name given to the Dalvik VM's process, until a shell script at /system/etc/init.d/zinit.sh is finished executing, if and only if the script exists (otherwise, zygote is launched as it always was).

    The solution coincides with another product called Xpose. The key to Xpose is also a modified version of the app_process binary, which provides additional "hooks" for customizing Android when the boot loader is locked, or a custom ROM is undesirable or unavailable.

    I noticed the key importance of the app_process binary just before I discovered Xpose, while perusing both init.rc and the AOSP source of Android 4.1.2_r1. This is a tiny file, and a deceptively simple source code. Xpose does not take advantage of the timing of the initialization of the Dalvik VM.

    I point all of that out because if you use Xpose you must choose a version of the app_process which is actually a fork of Xpose - so that you get BOTH of these features combined. Installing this solution would otherwise replace Xpose, disabling it. I've posted this with only app_process without Xpose as of this writing, but an Xpose version will appear later.

    The original version of the app_process source found at https://github.com/android/platform_frameworks_base/blob/master/cmds/app_process/app_main.cpp has this clause:

    Code:
    
        if (zygote) {
            runtime.start("com.android.internal.os.ZygoteInit",
                    startSystemServer ? "start-system-server" : "");
        }
    
    This is what starts the Dalvik engine. This code hasn't changed since 4.0.3 or before, and is very similar to code going back to 1.6 and earlier. This clause is run only once at power on, or on a soft boot if zygote dies. Xpose does not alter this clause. However, I alter it something like this:

    Code:
    
        if (zygote) {
    
            if ( access( "/system/etc/init.d/zinit.h", X_OK ) == 0 ) system( "/system/etc/init.d/zinit.h" ); 
    
            runtime.start("com.android.internal.os.ZygoteInit",
                    startSystemServer ? "start-system-server" : "");
        }
    
    
    This simple system call, executed on the condition that the script zinit.sh exists and has execute permission, is the entire modification made to app_process under this proposal (Xpose does other modifications). It is therefore conditional - that is, app_process works exactly as always if the script doesn't exist. Readers familiar with C may recognize known objections to the use of access, favoring stat - but they are concerns based on irrelevant security issues in this context. The validity of the objection is the fact that access uses a view of the file's permission in a way that differs from functions which actually open the file. It's possible these two permission profiles differ. However, in this use, the code is running as super user (even if the phone were not rooted, obviously after install) - so the notion is entirely irrelevant. stat would report the same view here, while access is smaller.

    What's important is that zygote will NOT be initialized until after the script completes (if it's executed at all), giving us the opportunity to reconfigure a great deal of the operating system without disturbing Dalvik, or Google's Gapps, or anything else known to us as Android.

    Installation is merely:

    Code:
    
    mount -o remount,rw /system
    cp /storage/external_SD/app_process_non_xpose_4.1.2 /system/bin/app_process
    busybox chmod a+x /system/bin/app_process
    mount -o remount,ro /system
    
    
    Where /storage/external_SD/app_process_non_xpose_4.1.2 is the modified binary provided for non Xpose compatability (or one of the other versions I'll be appending via links at the tail of this post). Any means of copying app_process into /system/bin is fine.

    The remount is required when /system is mounted read only (the default after boot).

    Obviously, busybox was assumed in this example...the default version of chmod doesn't support a+x, it takes only octal parameters.

    Now, astute readers immediately realize this has nothing to do with storage.

    That's correct, this only provides a hook by which we now have the ability to execute script(s) before Dalvik starts using /data. Consider this the installation of part 1 of the solution, the first pre-requisite to making this possible. The key is the ability to use mount while Dalvik is not initialized, which is a condition entirely out of reach unless we can unlock the boot loader reliably.

    Now we have to have something to mount. The typical external SD card is formatted in the vfat filesystem, otherwise known as FAT32. Unfortunately vfat doesn't support all of the identities and security data required by Android. For that we must use a linux compatible file system. You could choose ext2, ext4 or whatever you like which is linux compatible, and is supported by Android.

    Even many regarding themselves as expert may need reminding that the examples of commands which follow purposely erase the entire SD card. The word "backup" does not suffice to explain what is required if there is so much as 1 byte of material you must preserve. Transfer is, perhaps, a more suitable word. The recipient which stores the content of the SD card you may be re-partitioning will be, for a while, the primary source for that material. If there's anything important, it should be duplicated in a physically separate device. Many may not recognize what hidden files/directories are on the external card. These are commonly given names preceded with a ".", which are only visible when specifically directed for display (that is, ls -la instead of ls -l, or directing Linux GUI file browsers to show all files, including hidden files).

    Of distinct importance, however, are the hidden virtual filesystem which may appear on the SD card. These are uniquely invisible. Typically these appear as the result of using "move to phone" or other applications which relocate applications to the SD card. These are difficult to backup correctly. The unfamiliar should not apply. Personally, I just reserve the apk's of applications so that re-installation is trivial.

    This is exacty why I warned earlier that this is not a tutorial or instruction, but an engineering discussion on how this can be done. If this content is familiar to you (and most Android developers and Linux/Unix users are), this isn't a problem.


    At this point it is assumed that the SD card has no content of importance, as it will be completely eradicated.

    You can choose whatever method you prefer to partition the SD card, but from my notes this is a general outline of doing this at the command prompt in a shell, accessed via adb with root access:

    On the stock device, observe the mount command (type mount with no parameters).

    You'll notice that the external SD card is mounted as a vold device with major/minor numbers 7, 33.

    Code:
    ls -l /dev/block
    
    Reveals that /dev/block/mmcblk1 also is 7,33. This is the BASE to the external SD card on the F6 (other devices may differ). When partitioned, the first partition will be mmcblk1p1, partition two (fashioned automatically when present) will be mmcblk1p2, and so on. They appear automatically when the card is partitioned, or are entirely absent if the card is not partitioned.

    The external card must be unmounted

    Code:
    
    unmount /storage/external_SD
    
    
    At one point this was refused, so I had to

    Code:
    
    mount -o remount,rw /
    rm /dev/block/fuse
    unmount /storage/external_SD
    
    
    Fuse will be re-created on reboot, so that's actually harmless.


    I used

    Code:
    
    fdisk /dev/block/mmcblk1
    
    
    The SD card can't be partitioned if it's mounted - the block device is busy when it's mounted, Linux will stop you.

    I'll leave operation of fdisk to your own Google research. You might prefer an app like Aparted or something. If you use fdisk, be certain you use the "v" command to verify the partition configuration you choose, and use the "w" command to write that to the drive. If you don't "w" (write), it doesn't change the partitions, and if you don't "v" (verify) you can generate an invalid configuration.

    If you're disappointed that I don't detail the use of fdisk, then these notes and this project is not yet ready for you to experiment. There will likely be a version, later, more suitable where scripts may do this work for you.

    I personally decided upon a 4Gbyte partition and a 26Gbyte parition as partitions 1 and 2 respectively. fdisk automatically creates linux filesystem partitions (type 83). One most manually chose to edit the partition type and choose Win95 FAT32 to create a vfat partition.

    This configuration gave me the feature of having an automatically mounted external SD partition of 4Gbytes, while providing a 26 GByte partition for use as a source for the new /data directory.

    When all is finished I have 24+ Gbytes available after the basic applications plus about 6 games and few others installed, without any need for app by app fidling with storage configuration or concern about where downloads are going.

    In earlier experiments I chose one 32Gbyte partition instead. That works, but the phone behaves as though there is no external SD card (because that requires vfat to work), and occasionally a message appeared indicating the card seemed damaged (because it was expecting vfat).

    Creating the partitions doesn't complete the task, though.

    You have to create the filesystems, analogous to formatting a disk.

    On my 2 partition arrangment I have 3 devices routing to 7, 33:

    /dev/block/mmcblk1
    /dev/block/mmcblk1p1
    /dev/block/mmcblk1p2

    The first is the base....you leave that alone after fdisk has done it's work.

    The second, mmcblk1p1, should be the vfat partition (it's where the LG expects to find it). So, make the filesystem for it with

    Code:
    mkfs.vfat /dev/block/mmcblk1p1
    
    Then, depending on the filesystem you prefer, start with
    Code:
    mkfs.ext2 -m 1 -LExtData /dev/blockmmcblk1p2
    
    the -m 1 is "reserve 1% for superuser". You might have reason to use 2 or 3...Google for research if you need.

    The -L creates the label, in this case ExtData, for the filesystem. It's not an important point, your choice prevails.

    Now, if you want to use ext4 you can...but there is no mkfs.ext4. For ext4 you must turn the ext2 filesystem into an ext4 filesystem. I left ext2 on mine, it's fully compliant with Android's requirements, but to make this ext4 you use tune2fs. Google will provide the details, but an example command is:

    Code:
    tune2fs -O extents,uninit_bg,dir_index,has_journal /dev/block/mmcblk1p2
    
    Now, at this point the partitions are ready for mounting. In my own F6, where mmcblk1p1 is the vfat I mount thus:

    Code:
    
    mount -o rw -t vfat /dev/block/mmcblk1p1 /storage/external_SD
    
    
    This is the vfat partition mounted over the same position as the stock external SD card in a stock device. This mount is performed manually during the work, but upon reboot this partition is mounted automatically by the OS, because it is partition 1 and it is in vfat format. If either of these two points are not true, it will NOT automatically mount during boot.

    The second partition of my F6 is mounted thus:

    Code:
    
    mount -o remount,rw /
    mkdir /storage/sdcard_partition2
    mount -o remount,ro /
    mount -o rw -t ext2 /dev/block/mmcblk1p2 /storage/sdcard_parition2
    
    
    A little explanation:

    Every boot, the /storage directory is RESTORED to contain only two directories, sdcard0 and external_SD card.

    Every boot, therefore, a directory for the second partition must be created (it will not automatically mount). /storage is "owned" by the root directory, which is mounted read only. It must be remounted read write in order to create the directory. That's what the "remount,rX" commands are about.

    On my card I chose the ext2 filesystem. If you used tune2fs to create an ext4 system, or you want to use yaffs2, your parameter following -t will differ. Google provides, but again I remind readers if this is Greek, you're still just touring the experiment.

    Some may inquire, why not just wait and mount /dev/block/mmcblk1p2 over /data when the time comes? That works, and it is more "natural". However, in my own intent, I want access to the storage of the larger ext2 partition for other purposes (whatever that may be in the future), which means I intend to use a bind mount from a directory on the ext2 partition to serve /data.

    Ok, the stage is set...we're almost ready for the play.

    In manual experiment I created:

    Code:
    
    mkdir /storage/sdcard_partition2/.remounts
    mkdir /storage/sdcard_partition2/.remounts/data
    
    
    This creates a directory which will hold the original contents of /data, and eventually serve (ongoing) all purposes of the original /data, but with larger storage (26Gbytes in this example).

    I'll leave the appropriate chmod for you to choose, or to Google if you must...but if chmod is foreign to you, wait for a more public release.

    The original /data may be of use, for several reasons, but it will eventually be left alone ongoing until some reason comes that I should boot the phone without the external SD card. It can't be unmounted while the phone is running, and mounting over it with empty contents is the same result.

    Copying /data to the external card is as simple as:

    Code:
    
    cd /storage/sdcard_partition2/.remounts/data
    cp -rp /data/* ./
    
    
    Which takes a minute or so, depending on the size occupied, but it's not a good idea while in a shell. It will work, and maybe suffice, but Android is running. Google has it's databases open, tinkering around. Whatever we copy is going to be read while in use, possible creating SOME databases with invalid content. Most (maybe 98% or more) will copy just fine, but in my experiments Google was left puzzled when booted with a copy made this way. It eventually settles, but to me it's best to make this copy during the newly created opportunity of running scripts in Linux while Dalvik is suspended from initiating.

    The idea is to drop some scripts into position, intiated from /system/etc/init.d/zinit.sh. That script, whatever is in it, will be executed by the new app_process binary just before zygote (the Dalvik engine process) is launched.

    So, a one time installation script is in order, followed by a script left in place which operates this configuration from then on.

    Further, it is important that we should be able to boot the phone in a "stock" fashion at will...that is, with the original 1.27Gbyte partition servicing /data...from mmcblk0p15.

    Also, we may have good reason to review the content of the old friend now and then.

    So, here's one of the reasons I wanted to mount a directory in the second external partition rather than mount the partition over /data...zinit.sh will call a script from /storage/external_SD/.bootscripts/external_boot.sh, or something to that effect.

    The idea is that when zinit.sh fires on boot, if the external card is present, the script umounts /data from the interal source, mmcblk0p15, then mounts /data from the external card at /storage/external_sd/.remounts/data over /data. If the external card is NOT present, or a different card without a script at /storage/external_SD/.bootscripts/external_boot.sh is inserted, then the original /data is left mounted from mmcblk0p15 and the phone operates as a stock device.

    The minor catch being that the script in /system/etc/init.d/zinit.sh must check to see if there's a partition at /dev/block/mmcblk1p2 (in my example configuration), and that it mounts correctly as an ext2 filesystem (or whatever fileysystem you prefer, like ext4 or yaffs2), and then check to see if a script is found at /storage/external_SD/.booscripts/external_boot.sh before attempting to execute it.

    The script in /system/etc/init.d must, therefore, be fashioned according to your chosen configuration of the external SD card.

    While I suppose some mastery of sed and awk could fashion this script from source text (for a proper "consumer" level installer), I'd personally rather use C++ - but that's for later in the project, not this post.

    So, imagining for a moment that we're able to fashion a script that runs before Dalvik is active, I explore a few commands which would be of use in such a script.

    The first being:

    Code:
    unmount /data
    
    mount -o remount,rw /
    mkdir /storage/original_data
    mount /dev/block/mmcblk0p15 /storage/original_data
    mount -o remount,ro /
    
    
    This block is likely to be a part of that which performs the creation and mount of /storage/sdcard_partition2 mentioned above.

    At which point the original content of /data, as viewed on the stock running phone, is visible at /storage/original_data.

    The new proposed source for /data will be visible at /storage/sdcard_partition2/.remounts/data, while /data is actually empty at the moment.

    As such, for a one time configuration script during boot:

    Code:
    
    cp -rp /storage/original_data/* /storage/sdcard_parition2/.remounts/data 
    
    
    Would create a duplicate of the original data, with ownership and permissions duplicated, and links duplicated.

    This means that LG and the service provider's links into /system/.... for the pre-installed applications would already be correctly set.

    This leaves two interesting potential issues. There may be links created by link2sd or applications like it. They may have pointed to targets in /storage/external_SD/... or to hidden virtual filesystems the way "move to phone" does, depending on what application was used. If the 4Gbyte partition is restored with the old content of the external_SD, links would be restored, but the virtual filesystems are more difficult. I avoided the virtual filesystem problem by simply re-installing the applications, while the links I manually established earlier were served by restoring content to /storage/external_SD.

    When the copy is complete

    Code:
    
    mount -o rw,bind -t ext2 /storage/sdcard_partition2/.remounts/data /data
    
    
    supplies the needed /data directory from the external SD card's second partition, an ext2 filesystem in my example (adjust to suit your choice).

    The script which performed the copy above would then be removed, replaced with a script that just performs the mount over /data for subsequent boots.

    Additional scripts or an app may be required for special situations. The following describes what that's about.

    Configuring and using the device this way presents us with the following attributes:

    The original /data is known as /dev/block/mmcblk0p15, mounted as /storage/original_data. It will retain older content, but will not be updated when applications update.

    The "new" /data is served from /storage/sdcard_partition2/.remounts/data, is therefore visible from two perspectives (though .remounts is technically a hidden directory, it's very easy to get to it), and will receive application updates.

    This means there is an opportunity, on occasion, to review /data (as mounted from the external card), compare that to /storage/original_data, and copy updated content to it, including refashioning links that may have changed (should they even exist).

    If the phone is booted without the external card, the older versions of the applications would be used...then perhaps updated, if given the opportunity.

    If the script in /storage/external_data/.bootscripts/external_bootscript.sh is renamed, a reboot would leave the original /data (from /dev/block/mmcblk0p15) in place, operating on older content, but leaving the external card available - /storage/original_data would not exist, but /storage/external_SD and /storage/sdcard_partition2 would - or could - be mounted and useable, just not used by Dalvik (or Android).

    Now to some more experimental notes. I've not written the scripts described above, entirely (especially the copy of the /data partition). Instead, I did this:

    Code:
    sleep 7200
    
    In place of the external_boostscript.sh

    The reason was simple: I was (and still am) experimenting, and if you're considering it, you'll want this feature too.

    This sleep is in the external script, which means /system/etc/zinit.sh fired, mounted the external partition 2, but then slept for 2 hours.

    Zygote, and therefor the Dalvik runtime...everything we think of as Android...is suspended from initializing. The phone may be showing a black screen, or a boot logo...but otherwise seems stuck.

    Through adb, a shell allows me to manually arrange what I want for experiment.

    For example, I manually copied the original /data directory to the new partition on the SD card.

    I then manually unmounted /data (from /dev/block/mmcblk0p15, the original) and then mounted that duplicate from the external card over /data.

    When I'm ready to try it out, I then examine the ps report (for those not familiar, this is a list of all processes). From that I find the pid of the sleep which is hanging there, then kill it. The suspense is over. Android to boots (because zygote is no longer blocked by the sleep), but now in my new, manually arranged configuration (a one time experiment). A reboot returns things to "normal".

    Well.....normal after a 2 hour boot sleep. You'll want to use the shell to remove that script (this caught me twice).

    Further, it is not absolutely required that you format the SD card to experiment.

    If you've read this far you're probably going to try this at some point, even if you have to study a bit to figure out what each of these command lines are.

    So...how, if the SD card's stock vfat is incompatible with the concept of servicing /data?

    A virtual filesystem.

    Here's how.

    Make a directory on the external card....in the vfat system.

    cd to it.

    Now consider these notes.

    Code:
    dd if=/dev/zero of=filesys_data bs=32768 count=65536
    
    This creates a 2 Gbyte file on the sdcard called filesys_data, filling it with zeros. Alter count such that 32768 * count = size you want.

    This will be your virtual filesystem. It can evaporate at the mere "rm filesys_data".

    I discovered that on the F6 the maximum is 2Gbytes. VFAT is supposed to be able to handle 4Gbyte files, but on the F6, they evaporate on reboot. It works as long as you don't reboot, but there's some kind of bug involved. 2Gbytes is the maximum that actually worked for me. At least that's a little larger than the 1.27 Gbyte source, the size of /data.

    To use it, you need a node.

    Code:
    
    mknod /dev/block/filesys_data b 7 101
    
    This creates a node for a block device named filesys_data (you may substitute at will). Google for more info if you need. Think of it as mounting a socket on a wall to plug things into.

    Wire up the socket with losetup:

    Code:
    
    losetup /dev/block/filesys_data /storage/external_SD/whereveryouputit/filesys_data
    
    
    Adjust to your choices...whatever directory you create to store filesys_data (or whatever name you preferred). Losetup wires the socket to the source, in this case the blank file you created.

    Now, the socket refers to a blank fake disk, represented by the file filesys_data, in whereeveryouputit.

    We essentially plug in a piece of tech gear to prepare it with:

    Code:
    
    mkfs.ext2 -LExtData -m 1 /dev/block/filesys_data
    
    
    mksf.ext2 creates an ext2 filesystem (fully Linux compatible) using the socket (well...it's a node) /dev/block/filesys_data, which is wired to the blank file create whereeveryouputit/filesys_data.

    You can use tune2fs to make it ext4 if you prefer.

    It will be as large as the file you created ( bs * count ).

    It can now be mounted as if it were a separate SD card, formatted as ext2 (or 4 if you chose).


    Code:
    
    mount -o remount,rw /
    mkdir /storage/fake_data
    mount -o rw -t ext2 /dev/block/filesys_data /storage/fake_data
    mount -o remount,rw /
    
    
    We must have a directory to mount over, and we have to make it since /storage is always "restored" on boot, and since /storage is on a read only filesystem from root, we have to remount /.

    At this point /storage/fake_data is an ext2 (or ext4 if you made that) filesystem, hosted as a virtual disk created as a blank file on the SD card.

    Note, if you used tunefs to make this ext4, your -t parameter will be ext4 not ext2.

    This allows you to experiment WITHOUT having to reformat the SD card.

    You can now make directories inside /storage/fake_data....like .remounts/data...with which to experiment. It's a Linux filesystem, so it understands identities and permissions that vfat does not.

    Yes, you can copy all of /data into it and use THAT as the primary source for /data, as in the previous discussion. It will only be 2Gbytes, but it's larger than 1.2Gbytes and lets you prove this works before you commit to partitioning and re-formatting your SD card.

    This is closely related to what happens when you use the "move to phone" feature.

    Now for the rest of the story ;)

    I have this working, but there's a complaint....perhaps the last piece of the puzzle to solve.

    The device boots fine, runs fine...but it complains that identies on the /data directory are not "synchronized"....I believe this is simply an unattended detail as of this post. I'll continue studying tonight. It's a minor flaw, but has to be fixed (figured out) before it can be fashioned into a solution for a wider public. I just ignore the warning - the phone and all software, new installs - etc - work fine.

    This represents the state of the experiment as of Jan 24, 2014...it's a part time adventure.


    The binary for app_process version 4.1.2_r1, like the phone, which works from 4.0.3 through 4.2.2 (tested)...should work on 4.4.2, but I'll verify later, is at this link:

    http://www.ih-palladian.com/app_process_4_1_2_r1.gz

    BTW, if you didn't recognize that you have to gunzip that before installing it, you may not be ready to try this.

    If you read the above, you'll know what to do with it.

    Please post with experimental results and observations.
  17. ZipLipZ

    ZipLipZ Well-Known Member

    This looks really promising. Thanks for keeping the thread updated. I look forward to testing a beta version.
  18. JVene

    JVene Well-Known Member

    Thanks.

    Sunday I manage to complete the study (fixed that last, one unattended detail). I now have all the materials in place to make a clean, fully functional install of this technique. Aside from customizing the ROM, this is as close to an ideal result as can be managed.

    Now I'm going to prepare the "beta"....it may be an EARLY beta, but something most anyone with root access and ADB can handle.

    That should be in place by tomorrow.

    After I finally got this completely done, I did a hard reset on my phone, installed the solution (took about 10 minutes), then rebooted and proceeded to install all of the games and materials my son and wife wanted.

    At this point I have about 9 Gbytes of software installed...around 40 applications.

    Not once did I have to consider the storage issue.

    My 1.27 Gbyte partition, which is the one that fills up for everyone, isn't even mounted right now.

    The entire "Android" storage, the /data directory, is served from the external SD card from a 28 Gbyte partiion (I left 4 GBytes in vFat).

    This suddenly got real.

    Now I'll package for others to try
    o4tuna, infowarz, jjfad and 1 other person like this.
  19. jjfad

    jjfad Well-Known Member

    SWEET...
  20. lucasmun09

    lucasmun09 Well-Known Member

    YESSS!!! You finally did it! Congrats!!!!!!! But then what happens to that 1.2gb of partition..... maybe can't you combine it with other internal partitions that you discovered?
  21. JVene

    JVene Well-Known Member


    The 1.2Gbyte partition, mmcblk0p15, must stay as a "stock" configuration. The phone must be able to boot normally if the external card is removed. Factory resets with data wipes will direct their operation to that partition.

    It should be mountable as, say, /storage/internal_data, for review and even usage if there were really a reason, but I'll have to triple check it's availability for that.

    Easily one could erase the content of /storage/internal_data/dalvik-cache (the original, not the new, /storage/ext_part2/external_data. Doing that is of zero consequence if booting to stock is desired, because the cache is rebuilt automatically when required.

    The problem I see is that this partition must have enough space for the phone to operate if booted without the external card (or any other means of intentionally booting the stock configuration). Using it is hardly much benefit, but in my current state the partition has nearly 1 Gbyte available, because I did a hard reset before the "final" configuration of the phone. Put another way, if a hard reset sets the partition at 1 Gbyte free, like mine, the stock requirement is about 250 Mbytes. As long as that 250 Mbytes is intact, give or take the dalvik-cache content of perhaps 80 Mbytes, then that 1Gbyte should be available for user access.

    I don't think I'd touch the other partitions, because they're devoted to boot images, system images and other materials - including the /system directory which is still linked from the external_data source exactly as the stock configuration.

    I'm still dotting the "i"s and crossing the "t"s for a beta level installer. My current configuration was manually arranged according to the notes above with one simple final detail added.

    I had to be certain the ownership of /storage/ext_part/external_data was system:system. When it was manually created it was root:root, and that caused Android to complain about that. Attending that detail caused the phone to boot as if nothing had changed.

    It operates as if my son and wife had NOT objected to me soldering in a 32Gbyte replacement chip for the 4Gbyte internal one. Android has zero objection with the configuration.

    Depending on the day's events, there's a good chance I'll have something posted tonight, but it could slip a day.

    I'll need volunteers to test what looks like a 3 step process.
    o4tuna likes this.
  22. lucasmun09

    lucasmun09 Well-Known Member

    Hello! I don't know if you seen this thread before but I thought this may be very useful for future use. Its in the XDA Forums though..... Official bootloader unlock procedure from LG AND the one that works for everyone ;) - xda-developers

    The only usages that I know for unlocking bootloader are for custom roms and etc...... Maybe you can find more significant usage :D

  23. JVene

    JVene Well-Known Member

    Thanks for the link. I've found dozens of references for different approaches, and I had run across that one, too.

    Unlocking the bootloader would be a wonderful alternative to the modified app_process approach I use here, but unfortunately the F6 was introduced after that thread was initiated, so it doesn't universally apply.

    Loki can allow for booting a custom ROM, too, but like all such approaches it is dependent on specific versions and carriers. If the details are just right it does work. It exploits a vulnerability in the process for checking the signature during boot, interrupts it and loads any suitable ROM whether it passes the signature check or not. The problem is producing the image correctly. Loki happily generated an invalid image for my MetroPCS "i" ROM. Loki was expecting "e", but didn't complain (acted as though I had an "e").

    There are ways to finagle Loki into working on most F6 devices. My concern is 4.4.

    Loki has zero support for 4.4, and it could be a long while before it does, there's no way to know. Owners would have to be sure they block an automatic update, staying with 4.1.2 until Loki adds support, and then again it would be carrier/version specific ongoing.

    The modified app_process approach used here doesn't have such issues. It will work on 4.4.2, and can easily apply to any version, any carrier and any brand Android device. This applies only to the technique of modifying the configuration during boot during that critical (and very short) time period where Android is not yet active, but Linux is up and running.

    I would very much enjoy having an unlocked boot loader, but I would insist on that being a well supported, widely applicable solution independent of carrier and version. While the initial posts in that thread (and many others) promise a "universal" solution, that claim applied in early 2013. It doesn't anymore.

    From a more general standpoint, implementing a lock defeat approach is a LOT more work than creating and installing the alternative app_process binary, both for developers and owners.

    In all cases, once the ability to configure the operating system at the critical time synchronized point is established, the rest of what should be done is generally identical.

    That particular detail is already proven and solid here. My current time for this project is devoted to packaging the installation process for the average but slightly technically aware owner of the device, with a design that allows for expansion to other models and potentially any brand and OS version from 4.0.3 forward, even if the bootloader remains locked or the owner wishes to avoid using such exploits.
    o4tuna, Trevor Philips and jjfad like this.
  24. Trevor Philips

    Trevor Philips Well-Known Member

    Well, i reached all my internal storage today... sign me up for testing
  25. JVene

    JVene Well-Known Member


    Ok, let's deal with the pre-requisites, and I'll package some materials for you to automate the setup.

    Let me know if all of this is already true for your device:

    - rooted
    - adb and USB drivers installed and working
    - USB cable
    - full backup of data on SD card, including any hidden directories/files
    - full backup of all APK's (just in case).

    The last two are related. If you've used "move to phone", the APK's are stored in hidden filesystems on the SD card. In the event you don't have those backed up, you may have no other recourse after this but to install those again (but, from ADB, that's trivial).

    There is one thing to ponder. I've not experimented with doing this on a phone which has app2external_sd and/or link2sd (and it's cousins). I don't see this would be much of a problem, but here's what I do expect:

    The "swap" of internal / external storage will no longer be important, and may even be "in the way" after this change. I chose a configuration with 4Gbytes for the vfat partition (which still looks like an external card to Android, just 4 Gbytes in size)...if your adjustments with these two approaches put more than, say, 8Gbytes of material on the external card that could create a kind of space squeeze. In my view, while any space increase over the 1.27G is an improvement, I can see that we all really need at least 16Gbytes on Android devices, save for the very few.

    If you have a card reader on a PC, I do suggest you consider putting your external SD card on the PC for an image style backup using something like Acronis, EaseUS (which is free) or other software that can backup images without regard to disc layout, permissions or hidden directories - assuming you have material which may be valuable to you.

    I think you can use AParted (free) from the app store to partition the SD card. I've not used it, but if you're not familiar with the command line fdisk in Linux, that may be a good approach. You MIGHT be able to find something that could shrink your existing vfat partition, leaving space for a new partition sized to become your primary storage, in a Linux filesystem.

    If you find you can get all the way but not past the point of making a Linux filesystem on the second partition, that's fine.

    I'll prepare some scripts to automate the setup and post no later than Wednesday (likely Tuesday night).

    The scripts will assume there are two partitions on the external card. If you need help to get the partitions created let me know.

    Here's what you can look forward to based on my few weeks on the F6 with this working:

    My son wanted Need for Speed MW, Hot Pursuit AND Shift (he also wanted GTA and others, but they don't actually work well) - AND he's requested to try out Gangstar in 3 flavors. The three NFS have around 2 GBytes of data each when uncompressed. The download and install was effortless. Zero concern for where to put things. Android sees all of it, software and the (combined) 6 GBytes of data as residing on the primary storage (which it THINKS is the internal card - an internal card with 25+ Gbytes available).

    It REALLY works as though I've replaced the Internal card with a much larger one.

    By the end of the week I think we'll have around 16 Gbytes of application / game data on the device. That will leave me with only 10Gbytes available ( only :D ), but eventually I'm sure one of those games will be exhausted and removed.

    I may look into rumors that 64 GByte cards work.
    o4tuna, jjfad and Trevor Philips like this.

Share This Page