Thursday, April 26, 2012

Predicting Forex with Google preditct - Atempt #1

Since there lot of math involved in AI I was very happy to find that Google has product exactly for that. And to use it is relatively simple. My first attempt was to predict trading signals for Forex i.e. where to buy  (close buy) and to where sell (close sell). So I went through the price history and tagged time points as I would be actual trading on historical data. Here is an example:
Purple - SELL, yellow - CLOSE_SELL, green - BUY, blue - CLOSE_BUY
So after few days I had 44177 time points tagged with 5 different values: BUY, CLOSE_BUY, SELL, CLOSE_SELL and NONE. From all of these time points - 4724 were tagged with some signal and all others as NONE.
Then it was time to prepare prediction data. My learning data was similar to this:
SELL, 99 price values before time point, price at current time point
NONE, 99 price values before time point, price at current time point
NONE, 99 price values before time point, price at current time point
NONE, 99 price values before time point, price at current time point
....
CLOSE_SELL, 99 price values before time point, price at current time point
...
Model training took about 1.5 hours and Google Predict ended up with "classificationAccuracy": 0.9 
It was time to test the model.  To test the model I again use historical data. I sent about 60 request (some of them were from the same data I used to train model) and for all of then response was NONE. So I consider this as failure, but there is also bright side:
  1. Found a way how not to use AI and what learning data is considered bad;
  2. Learned how to use OAuth2 and how to use Google services from my code;
  3. Got experience with MongoDB (noSQL database);
  4. Refreshed my knowledge in Wicket, Swing and JFreeChart;
  5. It was fun and I got new ideas to test.

Wednesday, April 25, 2012

Google Predict - how to predict from your code using OAuth2

I had an idea to try Google Predict product to predict Forex. I will make another post about my attempt and in this post will be example of how to start using Google Predict and how actually access prediction from your code.
To get started please read this article. The article covers all the steps you need to to except the last one: how to get prediction in your code? While I was trying to do this I found Google libraries for this - don't try them - it is hard to understand them and they use old accessing framework OAuth1 which will be turned off soon. The good thing that Google also support new OAuth2 framework which is much more simple than ver1. I wrote an article about how easy is to use OAuth2 from any programing language.
Simple steps to get prediction:
  1. Create training data, upload it and train your model as described here.
  2. Get access token as described. Note: then creating URL to request user approve, in scope parameter include also 'https://www.googleapis.com/auth/prediction'
  3. Send you data sample to get prediction. The Java code is below.

try {
    String access_token="the token you received from Google from step 2";
    String modelId = "trained model ID you created in step 1";
    List<Object> values; //here should be all your data which will be sent to Google Predict to get prediction
    
    Scheme httpsScheme = new Scheme("https", 443, new EasySSLSocketFactory());
    SchemeRegistry schemeRegistry = new SchemeRegistry();
    schemeRegistry.register(httpsScheme);

    ClientConnectionManager cm = new BasicClientConnectionManager(schemeRegistry);
    HttpClient httpClient = new DefaultHttpClient(cm);

    HttpPost httppost = new HttpPost("https://www.googleapis.com/prediction/v1.5/trainedmodels/"+modelId+"/predict?access_token="+access_token);
    httppost.setEntity(new StringEntity(getRequestBody(values)));
    httppost.setHeader("Content-type", "application/json");

    log.debug("executing request to get token:" + httppost.getRequestLine());

    HttpResponse response = httpClient.execute(httppost);

    HttpEntity entity = response.getEntity();

    log.debug("response status:" + response.getStatusLine());
    ContentType contentType = ContentType.getOrDefault(entity);
    if (contentType.getMimeType().equals("application/json")) {
        //Here is reply from Google Predict
        String responseStr = IOUtils.toString(entity.getContent());
        log.debug("parsing json response:"+responseStr);
    }
    EntityUtils.consume(entity);
    httppost.releaseConnection();
    httpClient.getConnectionManager().shutdown();
} catch (Exception e) {
    log.error("Error retrieving prediction", e);
}


/*
Google predict expects input as this:
{
"input":{
  "csvInstance":[ col1_value, col2_value, ... ]
}
}
*/
private String getRequestBody(List<Object> values){
    StringBuffer sb = new StringBuffer("{\n\"input\":{\n\"csvInstance\":[");
    for(Object o:values){
        sb.append(o).append(",");
    }
    sb.deleteCharAt(sb.length()-1); //delete last coma(,)
    sb.append("]\n}\n}");
    log.debug("request body:"+sb.toString());
    return sb.toString();
}

 Now then you know basics you can read about other web services Google prediction support. Good luck predicting!

Tuesday, April 24, 2012

How to get nice and simple Date picker in Wicket

For my project in Wicket I needed simple JavaScript based Date picker. To get it - use Maven dependency:
<dependency>
    <groupId>org.apache.wicket</groupId>
    <artifactId>wicket-datetime</artifactId>
    <version>1.5.5</version>
</dependency>

Java Code:
import org.apache.wicket.extensions.yui.calendar.DatePicker;
import org.apache.wicket.extensions.yui.calendar.DateTimeField;


//Date field
DateTextField dateTextField = new DateTextField("dateTextField", new PropertyModel<Date>(
        this, "date"), new StyleDateConverter("S-", true));
DatePicker datePicker = new DatePicker();
datePicker.setShowOnFieldClick(true);
datePicker.setAutoHide(true);
dateTextField.add(datePicker);
add(dateTextField);

//Date and time fields
DateTimeField dateTimeField = new DateTimeField("dateTimeField", new PropertyModel<Date>(this, "date")){
    @Override
    protected boolean use12HourFormat() {
        //this will force to use 24 hours format
        return false;
    }
};
add(dateTimeField);

Html code:
<p>Date field: <input type="text" wicket:id="dateTextField" /></p>
<p>Date & time field<span wicket:id="dateTimeField"></span></p>

Result:
I have made another example of Wicket+JQuery date time picker, but this one I like more because it requires less code.

Monday, April 23, 2012

How to access Google services from your code (OAuth2)

Few days ago I wanted to access Google services from my own software. That was time consuming process because:
  • There are lot of samples, but they use OAuth1. This is more complex way and support will be turned off soon.
  • I was looking for some framework or libraries to help me. Of course I found, but they were hard to understand and again OAuth1 related.
So here is good news: you don't need  any specific framework or libraries to get using Oauth2. All Oauth2 communication is done in simple HTTP so it is usually GET or POST. Here are simple steps to get you on feet:
  1. Tell to Google what services you want to use in Google API console. From console you will get client_id, client_secret and redirect_uri parameters. 
  2. Construct URL and show it to your application user. By clicking this link user will be forwarded to Google and he will need to allow to use resources your applications requires.
  3. After user allows (or not) he will be forwarded back to your page (redirect_uri you speified in step 1). If user allowed access you - will receive code as URL parameter.
  4. Exchange code to access_token.
  5. Start using resources you asked passing  access_token.
Now then you know how everything works its time for code. By this time you can also try some things in Google OAuth playground.

Constructing URL:
public URI getLink() {
    List<NameValuePair> qparams = new ArrayList<NameValuePair>();
    qparams.add(new BasicNameValuePair("client_id", "your client id from step 1"));
    qparams.add(new BasicNameValuePair("response_type", "code"));
    qparams.add(new BasicNameValuePair("redirect_uri", "it should the the same as you filled for Google (case sensitive)"));
 
    //param scope - here you tell Google what services your application wants to access. In this example I want to access Gmail, User email and User profile
    qparams.add(new BasicNameValuePair("scope", "https://mail.google.com/ https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/userinfo.profile"));
    try {
        return URIUtils.createURI("https", "accounts.google.com", -1, "/o/oauth2/auth",
                URLEncodedUtils.format(qparams, "UTF-8"), null);
    } catch (URISyntaxException e) {
        log.error("Error creating URL", e);
    }
    return null;
} 

Exchanging code to access_token:
public Token getToken(String code) {
    try {
        Scheme httpsScheme = new Scheme("https", 443, new EasySSLSocketFactory());
        SchemeRegistry schemeRegistry = new SchemeRegistry();
        schemeRegistry.register(httpsScheme);

        ClientConnectionManager cm = new BasicClientConnectionManager(schemeRegistry);
        HttpClient httpClient = new DefaultHttpClient(cm);

        List<NameValuePair> qparams = new ArrayList<NameValuePair>();
        qparams.add(new BasicNameValuePair("code", code)); //The code google sent to application after user allowed access to his recources
        qparams.add(new BasicNameValuePair("client_id", "your client id from step 1"));
        qparams.add(new BasicNameValuePair("client_secret", "your client secret from step 1"));
        qparams.add(new BasicNameValuePair("redirect_uri", "your redirect uri from step 1"));
        qparams.add(new BasicNameValuePair("grant_type", "authorization_code"));

        HttpPost httppost = new HttpPost("https://accounts.google.com/o/oauth2/token");
        httppost.setEntity(new UrlEncodedFormEntity(qparams));

        log.debug("executing request to get token:" + httppost.getRequestLine());

        HttpResponse response = httpClient.execute(httppost);
        HttpEntity entity = response.getEntity();

        log.debug("response status:" + response.getStatusLine());
        ContentType contentType = ContentType.getOrDefault(entity);
        if (contentType.getMimeType().equals("application/json")) {
            String responseStr = IOUtils.toString(entity.getContent());
            log.debug("parsing json response:"+responseStr);
            return new Gson().fromJson(responseStr, Token.class);
        }
        EntityUtils.consume(entity);
        httppost.releaseConnection();
        httpClient.getConnectionManager().shutdown();
    } catch (Exception e) {
        log.error("Error retrieving token", e);
    }
    return null;
}

public class Token {
    String access_token;
    String token_type;
    Integer expires_in;
    Date requestedOn = new Date();

    //TODO: you need to add getters and setters
    
    @Override
    public String toString() {
        return String.format("requested_on:%s,token_type:%s,expires_in:%d,access_token:%s", requestedOn, token_type, expires_in, access_token);
    }
}

And finally - lets get User information:
public User getUserInfo(Token token) {
    try {
        Scheme httpsScheme = new Scheme("https", 443, new EasySSLSocketFactory());
        SchemeRegistry schemeRegistry = new SchemeRegistry();
        schemeRegistry.register(httpsScheme);

        ClientConnectionManager cm = new BasicClientConnectionManager(schemeRegistry);
        HttpClient httpClient = new DefaultHttpClient(cm);

        HttpGet httpget = new HttpGet("https://www.googleapis.com/oauth2/v1/userinfo?access_token=" + token.getAccess_token());
        log.debug("executing request:" + httpget.getRequestLine());

        HttpResponse response = httpClient.execute(httpget);
        HttpEntity entity = response.getEntity();

        log.debug("response status:" + response.getStatusLine());
        ContentType contentType = ContentType.getOrDefault(entity);
        if (contentType.getMimeType().equals("application/json")) {
            String responseStr = IOUtils.toString(entity.getContent());
            log.debug("parsing json response:"+responseStr);
            return new Gson().fromJson(responseStr, User.class);
        }
        EntityUtils.consume(entity);
        httpget.releaseConnection();
        httpClient.getConnectionManager().shutdown();
    } catch (Exception e) {
        log.error("Error retrieving User", e);
    }
    return null;
}

public class User {
    String id;
    String email;
    String name;
    String picture;
    String gender;
    String locale;

    //TODO: you need to add getters and setters

    @Override
    public String toString() {
        return String.format("id:%s,name:%s,email:%s,gender:%d,locale:%s", id, name, email, gender, locale);
    }
}


PS:
In my code I am using EasySSLSocketFactory which accepts all SSL certificates. This is OK for testing purpose, but you should definitely chance it in production.
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.http.conn.scheme.SchemeLayeredSocketFactory;
import org.apache.http.params.HttpConnectionParams;
import org.apache.http.params.HttpParams;

import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLSocket;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.Socket;
import java.security.SecureRandom;
import java.security.cert.X509Certificate;


public class EasySSLSocketFactory implements SchemeLayeredSocketFactory {
    private SSLContext sslcontext = null;
    private Log log = LogFactory.getLog(EasySSLSocketFactory.class);

    private static SSLContext createEasySSLContext() throws IOException {

        SSLContext context = null;
        try {
            context = SSLContext.getInstance("SSL");
            context.init(null, new TrustManager[]{new X509TrustManager() {
                public X509Certificate[] getAcceptedIssuers() {
                    return null;
                }

                public void checkClientTrusted(X509Certificate[] certs, String authType) {
                }

                public void checkServerTrusted(X509Certificate[] certs, String authType) {
                }
            }}, new SecureRandom());
        } catch (Exception e) {
            throw new IOException(e.getMessage());
        }
        return context;
    }

    private SSLContext getSSLContext() throws IOException {
        if (this.sslcontext == null) {
            this.sslcontext = createEasySSLContext();
        }
        return this.sslcontext;
    }

    public Socket createLayeredSocket(Socket socket, String s, int i, HttpParams httpParams) throws IOException {
        log.debug("createLayeredSocket");
        return getSSLContext().getSocketFactory().createSocket();
    }

    public Socket createSocket(HttpParams httpParams) throws IOException {
        log.debug("createSocket");
        log.debug(Thread.currentThread().getStackTrace());
        return getSSLContext().getSocketFactory().createSocket();
    }

    public Socket connectSocket(Socket sock, InetSocketAddress remoteAddress,
                                InetSocketAddress localAddress, HttpParams params) throws IOException {

        int connTimeout = HttpConnectionParams.getConnectionTimeout(params);
        int soTimeout = HttpConnectionParams.getSoTimeout(params);
        SSLSocket sslsock = (SSLSocket) ((sock != null) ? sock : createSocket(params));
        if (localAddress != null) {
            // we need to bind explicitly
            sslsock.bind(localAddress);
        }

        sslsock.connect(remoteAddress, connTimeout);
        sslsock.setSoTimeout(soTimeout);
        return sslsock;
    }

    public boolean isSecure(Socket socket) throws IllegalArgumentException {
        return true;
    }
}


To make things easier here are my Maven dependencies:
        
<!--  Apache HTTP commons  -->

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpcore</artifactId>
    <version>4.2-beta1</version>
</dependency>
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.2-beta1</version>
</dependency>
<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-io</artifactId>
    <version>1.3.2</version>
</dependency>

<!-- GSon -->

<dependency>
    <groupId>com.google.code.gson</groupId>
    <artifactId>gson</artifactId>
    <version>2.1</version>
</dependency>