[HOWTO] Get a opensource equivalent to MS synchronizations

Talk about Mozilla browsers for small devices such as PDAs and cellphones.
Locked
xryl669
Posts: 5
Joined: December 16th, 2005, 7:30 am

[HOWTO] Get a opensource equivalent to MS synchronizations

Post by xryl669 »

Hi all,

I'm writing this post here because I think it is the most convenient place to store such a post.
What I will show you here is:
1) - Speed up rendering by hugely decreasing the web bandwidth
2) - Get rid of active sync in favor to a standard based synchronization framework
3) - Remove the need for slow and (inaccurate) rendering in PIE
4) - Share your calendar/task/todo on the web
5) - Access your GMail account even on PPC2002

What is required:
1) - An internet visible computer you own. It can run Linux or Windows
2) - A pocket pc (but any other SyncML compatible devices will do the trick)
3) - (Not required, but it is way better if you have one) a free php based host service

Okay, let's start hacking:
Let the internet-visible computer be PVC, the pocket PC PPC, and the isp ISP.

1) Speed up rendering
This is what I miss the most when I don't have it. The idea is to set up a proxy on your computer that will clean up the web shit, compress the pictures and text and send the result back to your PPC.
In this field, I've tried maybe 99% of the available solutions, and I can tell you that there are only two or three competitors that worth wasting time with.
The best softwares are:
- WebCleaner
Very good compressing and cleaning proxy (by far the best I've tested so far)
Pros:
1) HTML and Javascript cleaning / compression
2) Image overcompression (conversion to JPEG)
3) Ad removal
4) Many authentications (now works with direct NTLM authentication
in IE and pocket IE)
5) Plugin based with regular expression white/black lists
6) Intuitive, can work with Squid

Cons:
1) Made with python (but works with psyco), so it consumes large
amount of memory to start (45MB)
2) Current version is still leaking memory so it needs to be restarted often
(however I've managed to add scheduled garbage collect calls that allows it to run on a 64MB machine during a day without restart)
3) Requires Python (which is not available everywhere)

and Ziproxy
- Ziproxy
Young software with interesting features
Pros:
1) No virtual machine, hence should be fast
2) HTML compression without selective input (doesn't accept
gzip stream!)
3) Image overcompression (conversion to JPEG)
4) Can do JP2K compression (hence more saving)
5) Basic authentication
6) As it is per connection-process based, will not leak memory (kernel takes care of cleaning)

Cons:
1) No HTML cleaning (require ungzipping content)
2) No ad removal (a simple regular expression parser on URL with a AdBlock
like list wouldn't require much work but would do wonders...)
3) No Javascript cleaning (this requires much work for almost no value except
pop up blocking and advanced ad removal)

There is also rabbit which uses a JVM, but it is way too buggy and slow.

So far so good, I've used Webcleaner for 3 month, until I switched to Ziproxy. I've added adremoval to ziproxy (which is available as a patch on the mailing list). The tutorial here will show you how to set up ziproxy. Ziproxy has the enormous advantage of not leaking nor requiring huge amount of memory (I only have 64MB on my PVC)

On the PVC, if it is running Linux, you will simply have to download Ziproxy from the CVS:

Code: Select all

cvs -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/ziproxy login
cvs -z3 -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/ziproxy co -P ziproxy-default

Then apply the adblocking patch (this patch add ad blocking to the proxy by using a website list file that is compatible with Firefox's AdBlock extension)

Code: Select all

cd ziproxy-default
wget http://crashreport.free.fr/add_adblocking.patch
wget http://crashreport.free.fr/HOWTO_adblocking
patch -p1 < add_adblocking.patch
./bootstrap
make
make install

Now you have a Ziproxy running. You will have to configure it by manually editing the ziproxy-default/etc/ziproxy.conf and copying it to /etc/ziproxy.conf.
You will also have to download an adblock url list (google => adblock list) and save it as adblock.list in the same directory as ziproxy.conf. Please follow the HOWTO_adblocking to set up Ziproxy correctly.

Then, you'll have to open a port on your firewall to the proxy (must be the same as the ziproxy port of course). Please refer to your firewall documentation for this.

Finally, test the proxy by using your PPC or another computer, and setting the proxy in the browser (for PC). Under Windows CE, you'll have to:
  1. Click on Start icon
  2. Click on setting
  3. Click on connections tab
  4. Click on Connections icon
  5. Create a My Work network, and click on "Manage settings..."
  6. Click on the "Proxy settings" tabs
  7. Check the "This network connects to the internet" and "This network uses a proxy" check boxes
  8. Click on "Advanced..."
  9. Click on "HTTP",
  10. Enter the public IP address of your PVC and the external port
  11. If you've set up authentication, enter your authentication information too. Leave "domain" blank
  12. Then, you have to tell your device to use the "Work network" connection. Follow 1,2,3,4 previous step again.
  13. Click on "Advanced" tab
  14. Click on "Select Networks"
  15. Change "My ISP" to "My work network" in the first box to use the proxy. Change it back to "My ISP" to resume you previous slow rendering speed.

If everything was done like I've written, you should enjoy a 3 or 4 times speedup in browsing the web (ever with PIE or Minimo).
Those steps works for PPC2002 PPC2003 and WM5.0.
If you want to use Windows for your PCV, use Webcleaner as you probably have 512MB or more of RAM. However, PM for a small patch I've made to Webcleaner that decrease the memory usage by 60% after running for an hour (even with this patch, Webcleaner leaks memory so you'll have to restart Webcleaner every 2 days or so).


2) Remove this nasty active sync
If you are like me, I've had trouble with the last active sync version (firewall issues). They I realized how dependant my personal data was to the active sync technology. I've looked for alternatives, and I've successfully found / setup synchronization without active sync.

The main issue with Microsoft is that they only follow a standard when they have money to earn from it. The world synchronization standard is based on SyncML protocol. However, until recently, there wasn't any SyncML client for PPC.

I've found an opensource SyncML client here. It worked out of the box for my PPC2003 and WM5.0, but required recompilation on my PPC2002 device. I will upload the PPC2002 version on demand. The main advantage of SyncML is the ability to synchronize with any SyncML compliant software (current SyncML client are Evolution, Mozilla Calendar, Mozilla Thunderbird, Kontact, KMail, etc...) through plugins.

To be able to synchronize with a SyncML client, you need a SyncML server. Even if OpenSync team say they have one, I never succedded in compiling and running it. So, finally, I've used the Sync4j server even if it is java based.

To summarize:

  1. Install a SyncML client on your PPC from here.
  2. Install a SyncML server on your PVC from here.
  3. Install the required plugin (I'm using Mozilla Synchronization plugin from here. It works on v0.2 only).
  4. Set up the SyncML server by running the Sync4j admin interface (default user is admin, and default password is "sm" or "s4" or "sc" depending on version)
  5. Launch the client on the PPC and write down the device ID
  6. Launch the calendar on the PVC or any PC, and write down the device ID in the SyncML server admin interface, create a user account with a password
  7. Create a 2 devices with the ID you've written
  8. Create a parternship between them (this is required)
  9. On the (PPC,Mozilla) SyncML client, enter the user login/password and PVC IP address and port you've just created in the option dialog
  10. Click on synchronize on any client, and it should say "Connection successful"
  11. Remove the guest/guest user (security reasons)
  12. (Optional)Configure the SyncML server port in the admin interface, and open it on your firewall if you want to be able to synchronize from the internet


3) Trash PIE
Okay, for this, you have 2 possibilities. Either you find a midlet virtual machine for your PPC and install opera mini (follow this url on your PPC, not on a PC).
You'll find "midlet manager" on xda-developers forum (google around). It works on my Qtek, but I've no idea how it will manage on other platforms. Opera mini is incredibly fast, and perform a one column layout rendering that rules.

The second possibility is to use either Netfront or Minimo. As I'm writing this, I would say that Netfront have as many issues as Minimo, but Minimo keeps improving each version.
I would have loved an embedded KHTML version, but it is not the right place to ask for ;-)


4) Share your calendar on the web
Okay, the idea here is to be able to access/modify your calendar from anywhere, and share it with others.
I suppose you're using Mozilla Calendar or Sunbird (if not, well, good luck).

If you don't have an ISP with PHP hosting service,
  1. Install apache on your computer (there are thousand of site about this)
  2. Install PHP on your computer (if you use Windows, download easyphp)
  3. Check that everything is working... (open ports in your firewall, authentications, etc...)

Then,
  1. Download and uncompress PHPiCalendar
  2. Modify the config.inc.php file to match your value (typically, you'll have to enable "internal" authentication and set up the locking mecanism for calendar access, see the wiki for that point)
  3. Upload PHPiCalendar files to your hosting service or in your PVC
  4. Upload an empty myCalendar.ics in the calendars subfolder
  5. In Mozilla, click on File/Subscribe to remote calendar
  6. Enter the name for this calendar (ie: MyCal)
  7. In the url enter something like :

    Code: Select all

    http://YourHOSTorYourPVCIPAddress/calendars/publish.mozilla.php/myCalendar.ics

    Please take care to use publish.mozilla.php/ and not publish.mozilla.php? in the URL
  8. Enter the authentication information
  9. Add a new event/task/todo item, and make sure you select the MyCal calendar
  10. If you've checked the auto-publish (and removed the readonly) check box, you can go to http://YourHOSTorYourPVCIPAddress/ on your browser, and you'll see your calendar.
  11. At any time, if you add a event/task/todo with private flag set (and not public), then PHPiCalendar will hide this event


5) Accessing GMAIL with a PPC2002
If you have a PPC2002, then you'll realize that it is not possible to access GMAIL, either with PIE, either with Outlook, either with Minimo (as far as I've tested).
However, you can download QMail from here. It is open source, and support SSL/TLS as required by GMail. The download link is here.
  1. Download the PPC2002 and OpenSSL libraries for PPC2002.
  2. Enable POP access in GMail (using your PVC).
  3. Create an account in QMail, and follow the Google instructions from here. The dialog box tell you what to set up on QMail
  4. Then, you'll have to add the Google certificate for allowing the connection so
  5. Download this file
  6. Unzip it, it contains "ThawteServerCA.txt" that should be renamed to "ca.pem" and saved in QMail Mail/Security folder on the PPC.
  7. Use port 995 and SSL for incoming, and port 587 STARTTLS for outgoing.
  8. You'll find more information here


6) To conclude
By using the right standards, you can easily set up a true PIM for 0 cents, with open source code as a bonus (hence, you'll never be dependant on a vendor software anymore).
I've loosed my data so many times before (out of power most of time), that I can surely say that being able to synchronize via the internet is the holy graal of data survival.
Being able then to allow any of my friend to see what I've planned in realtime is very very interesting.
At the same time, saving time while browsing by compressing the bandwidth by a 3x factor should be the minimum requirement on any embedded browser. It's like having an xDSL bandwidth on a old 33.8kpbs modem.

I hope it helps, Cyril
gils
Posts: 5
Joined: January 25th, 2006, 6:58 pm

Re: [HOWTO] Get a opensource equivalent to MS synchronizatio

Post by gils »

[*]Create a 2 devices with the ID you've written
[*]Create a parternship between them (this is required)
[*]On the (PPC,Mozilla) SyncML client, enter the user login/password and PVC IP address and port you've just created in the option dialog


3 questions:
-The calendar plug-in device has the same ID as one already existing in Sync4j. Should I use the existing one, create another/delete the existing?
-I could not find any partnership in Sync4j. How do you create one? What menu?
-IP Address: I have my PDA plug on USB to my computer. Sould I put "localhost" or my real IP address?

I tried a lot of settings but it still doesn't work.
Thanks for any help.
Gilles
lupho
Posts: 1
Joined: January 28th, 2006, 11:57 am

Re: [HOWTO] Get a opensource equivalent to MS synchronizatio

Post by lupho »

Hi, I have not so much experience, but problems, too - but I like the idea of it and would be glad if somebody could help

[*]Create a 2 devices with the ID you've written
I think we should use the existing one
[*]Create a parternship between them (this is required)
I think this means the creation of principals (see manual sync4j)
[*]On the (PPC,Mozilla) SyncML client, enter the user login/password and PVC IP address and port you've just created in the option dialog
plug my PDA on USB to my computer
-> that's what I want, too - but no idea: put "USB" to "port"?? My mobile phone tried to connect to the internet at this step when I used localhost.

Thanks for any help
xryl669
Posts: 5
Joined: December 16th, 2005, 7:30 am

Post by xryl669 »

3 questions:
-The calendar plug-in device has the same ID as one already existing in Sync4j. Should I use the existing one, create another/delete the existing?
-I could not find any partnership in Sync4j. How do you create one? What menu?
-IP Address: I have my PDA plug on USB to my computer. Sould I put "localhost" or my real IP address?

The calendar plug-in should have a unique ID. I recommend you remove the existing one and create a new one.
Then, in Sync4j server console (called SyncAdmin), you'll have to create a partnership (called Principal in Sync4j)
Please refer to http://download.forge.objectweb.org/syn ... -2.3b3.pdf page 20
If you are using the lastest ActiveSync, then your PDA is viewed like a new network interface (available in Control Panel/Network connections...). To know if the IP link works, press Window Key + R, cmd.exe, ipconfig
If you see that the PDA's network connection get an IP, try to ping it and the PDA's IP (the PDA's IP is visible in Settings tab/Connections/Network adapters of the PDA OS). If the PDA's IP is 0.0.0.0 you have to solve activesync problem by yourself.
If it answers the ping, then write down the IP of your PC (the one shown in ipconfig command output).
In Sync4j PDA client, you've have to write the PC'IP in the configuration setting (try this first, and if it doesn't work, you can put the PC's network name found in network tab when pressing Window Key + Pause)

[*]On the (PPC,Mozilla) SyncML client, enter the user login/password and PVC IP address and port you've just created in the option dialog
plug my PDA on USB to my computer
-> that's what I want, too - but no idea: put "USB" to "port"?? My mobile phone tried to connect to the internet at this step when I used localhost.


You have to know that the USB connection between your PDA and PC is now emulating an ethernet connection.
So your both your PDA and PC will be granted a specific IP address.
For example, on my PC, Active Sync (4.1) can't succeed any connection. However, I've installed it because it provides a USB driver that, when the PDA is cradled, adds a (virtual) network interface between my PC and my PDA. The IP link does work (using the default IP given in ActiveSync) even if activeSync doesn't.
If you are using Linux, this driver is called usbnet.

Once the IP link is set up correctly (the IP base, ie xxx.yyy.zzz. of the IP address, should be the same on both devices, only the last number should be different), I'm able to ping the PDA from the PC (hence the communication is working). Of course the IP base must be different from your existing network card (don't use the same mask as your LAN card etc...). You can change IP address with the setting / control panel "Network/connection" icons.

Synchronization then is a breeze with Sync4j.

Once my IP configuration is good (working), I usually write it down, and uncheck the "automatic get an IP address" and enter the good configuration, as it breaks thing more than it solves problems (for example, if you use another USB port, you'll get another IP for both your PC /PDA !).
Locked