Ivan Idris Blog

September 5, 2010

Can we predict stock prices with Grails Finance?

Filed under: programming — Tags: , , — admin @ 8:32 pm

Grails Finance 0.8

Yes, we can. Not always correctly, of course. One way to predict things, is to try to find the probability distribution for a phenomenon. Usually a distribution is charted as a histogram, so that is what I did. I have a lot of ground to cover today, so I will be skipping some details here and there.

Services

I went a bit overboard with all kinds of Grails services. These are the services I made

  • HistoricalQueryService – retrieves historical data from the database.
  • FrequenceService calculates frequencies.
  • SubtractService – subtracts vectors and calculates deltas (current – previous day).
  • LnService – calculates ln var and ln var – ln var previous day.

Historical query service

The job of the historical query service is to get from the database historical data based on a symbol or symbols and a field name. For instance, DIA and Close. When two symbols are specified the service matches the data of the two instruments based on the added timestamp and subtracts the data. Here is the code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

    ...
    def queryVals(symbol,fieldName) {
       def instrument = Instrument.findBySymbol(symbol)
       def field = Field.findByName(fieldName)
       def fieldVals = FieldValue.findAllByInstrumentAndField(instrument, field)
       def data = []
 
      fieldVals.each {  data << Double.valueOf( it.val )}
 
      return data
    }
 
    def queryPairVals(symbol, symbol2, fieldName) {
    def instrument = Instrument.findBySymbol(symbol)
    def instrument2 = Instrument.findBySymbol(symbol2)
    def field = Field.findByName(fieldName)
    def fieldVals = FieldValue.findAllByInstrumentAndField(instrument, field)
    def fieldVals2 = FieldValue.findAllByInstrumentAndField(instrument2, field)
 
    def data = []
 
   for(int i = 0; i < fieldVals.size() && i < fieldVals2.size(); i++) {
      def sameDateVal = fieldVals2.each { 
   if(fieldVals[i].added == it.added) {
      return it.val
            }
         }
 
         if(sameDateVal == null) continue   
 
         data << Double.valueOf(fieldVals[i].val) - Double.valueOf(sameDateVal[0].val)
      }
 
      return data
    }
    ...

and here is the test code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

...
    void testQueryVals() {
      def type = new InstrumentType(type:"type", description:"desc")
      assertNotNull "Type should not be null", type.save()
 
      def ds = new Datasource(name:"DASOURCE", description: "desc")
      assertNotNull "DS should not be null", ds.save()
 
      def ins = new Instrument(name: "TEST", symbol: "TEST", 
         source: ds, instrumentType : type)
 
      assertNotNull "Instrument should not be null",ins.save()
 
      def f = new Field(name:"TOAST", description: "desc")
      assertNotNull "f should not be null", f.save()
 
      def v = new FieldValue(instrument:ins, field:f,
         added: new DateTime(), val: "1.0")
      assertNotNull "V should not be null", v.save()
 
      def vals = historicalQueryService.queryVals("TEST", "TOAST")
      assertNotNull "Vals should not be null",vals
      assertEquals "There should be 1 value", vals.size() , 1
    }
 
   void testQueryPairVals() {
      def type = new InstrumentType(type:"type", description:"desc")
      assertNotNull "Type should not be null", type.save()
 
      def ds = new Datasource(name:"DASOURCE", description: "desc")
      assertNotNull "DS should not be null", ds.save()
 
      def ins = new Instrument(name: "TEST", symbol: "TEST", 
         source: ds, instrumentType : type)
      def ins2 = new Instrument(name: "TEST2", symbol: "TEST2", 
         source: ds, instrumentType : type)
 
      assertNotNull "Instrument should not be null",ins.save()
      assertNotNull "Instrument should not be null",ins2.save()
 
      def f = new Field(name:"TOAST", description: "desc")
      assertNotNull "f should not be null", f.save()
 
      def now = new DateTime()   
      def v = new FieldValue(instrument:ins, field:f,
         added: now, val: "1.0")
      def v2 = new FieldValue(instrument:ins2, field:f,
         added: now, val: "3.0")
      assertNotNull "V should not be null", v.save()
      assertNotNull "V2 should not be null", v2.save()
 
      def vals = historicalQueryService.queryPairVals("TEST", "TEST2", "TOAST")
      assertNotNull "Vals should not be null",vals
      assertEquals "There should be 1 value", vals.size() , 1
   }
...

Frequence service

The frequence service defines a number of bins and counts the number of occurrences in each bin. I defined a Bin class as follows

1
2
3
4
5
6
7
8
9
10
11
12
13

...
   double start
   double end
   double mid
   String label
 
   Bin(s, e) {
      start = s
      end = e
      mid =  (start + end) / 2 
      label = sprintf('%.3g', mid)
   }
...

The service code is shown below. First I calculate the size of bin, then I make a list of Bin objects. After that I check in which bin each input value falls and increase the count for the lucky bin. There is also a convenience method that reduces the x labels by a factor of million – this is handy for volumes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

...
    def frequence(input) {
      def step = calcStep(input)
      def bins = makeBins(input.min(), step) 
      def frequency = [:]
 
      for( int i in 0..<NBINS) {
         frequency[ bins[i].label ] = 0
      }
 
      def size = input.size()
 
      input.each { 
         def index = calcIndex(it, bins)
         frequency[ bins[index].label]++
      }   
 
      return frequency
    }
 
   def makeBins(min, step) {
      def bins = []
 
      for( int i in 0..<NBINS) {
         bins << new Bin(min + i * step, min + (i + 1) * step)
      }
 
      return bins
    }
 
   def reduceKeys(frequency) {
      def reduced = [:]
 
      frequency.each { 
         key,value -> 
            reduced[Math.round(Double.valueOf(key)/(1000 * 1000))] = value
      }
 
      return reduced
   }
 
   def calcIndex(it, bins) {
      for(int i in 0..<bins.size()) {
         if(it >= bins[i].start && it < bins[i].end) {
            return i
         }
      }
 
      return bins.size() - 1 
   }
 
   def calcStep(input) {
      return (input.max() - input.min())/ NBINS
   }
...

Here is some of the test code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

...
   void testFrequence() {
      assertNotNull frequenceService
      def frequencies = frequenceService.frequence([1.0, 1.5, 32.0])
      assertNotNull frequencies
      assertEquals "Number of bins incorrect", 30, frequencies.size()   
 
      def store = 0
 
      frequencies.each { key,value -> store += value }
 
      assertEquals 3, store
    }
 
   void testStep() {
      assertEquals 1.0, frequenceService.calcStep([0,30.0])
   }
...

Subtract service

The subtract service subtracts vectors and calculates deltas (current – previous day).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

...
    def minPrev(array) {
      def subtracted = []
 
      for(int i in 1..<array.size() ) {
         subtracted << array[i] - array[i - 1]
      }
 
      return subtracted
    }
 
   def min(left, right) {
      def subtracted = []
 
      for(int i in 0..<array.size() ) {
         subtracted << array[i] - array[i - 1]
      }
 
      return subtracted
   }
...
   def testMinPrevious() {
      def subtracted = subtractService.minPrev([1, 2, 3])
      assertNotNull subtracted
      assertEquals 2, subtracted.size()
      assertEquals 1, subtracted[0]
      assertEquals 1, subtracted[1]
   }
...

Ln service

The Ln service calculates the natural logarithm of a variable vector and the difference of the ln of a variable and the ln of the previous variable value. There is a small problem with logarithms. You cannot take the logarithm of 0 or negative numbers. So I cheat a little by substituting negative numbers by their absolute value and avoiding 0.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

...
    def ln(numbers) {
      def result = []
 
      numbers.each {
         if(it != 0) 
            result << Math.log(Math.abs(it))
       }
 
      return result
    }
 
    def deltaLn(numbers) {
      def result = []
 
      for(int i in 1..<numbers.size()) {
         if(numbers[i] != 0 && numbers[i - 1] != 0)
            result << Math.log(Math.abs(numbers[i])) - Math.log(Math.abs(numbers[i-1]))
      }
 
      return result
    }
...

Some more tests

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

...
    void testLn() {
      def outcome = lnService.ln([-1,0, 1])
      assertNotNull outcome
      assertEquals  2, outcome.size()
      assertEquals  0, outcome[0]
    }
 
   void testDeltaLn() {
      def outcome = lnService.deltaLn([-1,0,1,1])
      assertNotNull outcome
      assertEquals  1, outcome.size()
      assertEquals  0, outcome[0]
   }
...

Controller

Services on their own don’t do much, so we need a controller to use the services and communicate with the views.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

...
   def calcFrequence(vals, fieldVar) {
      def frequence = frequenceService.frequence(vals)
 
      if(params.fieldVar == 'Volume') {
         frequence = frequenceService.reduceKeys(frequence)
      }
 
      return frequence
   }
 
   def calcDeltaFrequence(vals) {
      def subtracted = subtractService.minPrev(vals)
      return frequenceService.frequence(subtracted)
   }
 
   def calcLnFrequency(vals) {
      def lned = lnService.ln(vals)
      return frequenceService.frequence(lned)
   }
 
   def calcDeltaLnFrequency(vals) {
      def deltaLned = lnService.deltaLn(vals)
      return frequenceService.frequence(deltaLned)
   }
 
   def makeResponse(vals) {
      def frequence = calcFrequence(vals, params.fieldVar)
 
      def deltasFrequency = calcDeltaFrequence(vals)
 
      def lnFrequency = calcLnFrequency(vals)
 
      def deltaLnFrequency = calcDeltaLnFrequency(vals)
 
      return [data : frequence, 
         deltaData : deltasFrequency,
         lnData : lnFrequency,
         deltaLnData : deltaLnFrequency]
   }
 
    def index = { 
      if(params.fieldVar != null) {
         def vals = historicalQueryService.queryVals( params.select1, params.fieldVar);
         return makeResponse(vals)
      }
   }
 
    def pairs = { 
      if(params.fieldVar != null) {
         def vals = historicalQueryService.queryPairVals( 
            params.select1, params.select2, params.fieldVar);
         return makeResponse(vals)
      }
   }
...

Views

There are 2 views – the main view for a single symbol and a view for a pair of symbols.

Main view

The view consists of a form with 4 histograms beneath it. The form lets you choose a symbol and a field. The histograms are made with the Google Charts Grails plugin.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96

<html>
   <head>
      <title>Historical data histogram</title>
   </head>
   <body>
      <table>
         <tr>
            <g:form>
            <td>
                  <g:select name="select1" from="${['DIA', 'SPY', 'GLD']}" value="${params.select1}"/>
            </td> 
            <td>
               <g:select name="fieldVar" from="${['Open', 'High', 'Low', 'Close', 'Volume']}" value="${params.fieldVar}"/>
            </td>
            <td>
               <g:actionSubmit name="doChart" action="index" value="Chart"/>
            </td>
            </g:form></td>
         </tr>
      </table>
 
      <g:if test="${data != null}">
           <g:barChart type="bvs"
                title="Histogram ${params.select1} (${params.fieldVar})"
            size="${[900,200]}" 
            colors="${['00FF00']}" 
            fill="${'bg,s,efefef' }"
            axes="x,y"
            axesLabels="${
               [0:data.keySet(),
               1:[0,data.values().max()/2, data.values().max()]
               ]
                }"
            dataType="simple"
            data="${data.values().asList()}"
            />
      </g:if>
 
      <br/> <br/>
 
      <g:if test="${deltaData != null}">
           <g:barChart type="bvs"
                title="Histogram ${params.select1} (${params.fieldVar} - previous day ${params.fieldVar}) "
            size="${[900,200]}" 
            colors="${['00FF00']}" 
            fill="${'bg,s,efefef' }"
            axes="x,y"
            axesLabels="${
               [0:deltaData.keySet(),
               1:[0,deltaData.values().max()/2, deltaData.values().max()]
               ]
                }"
            dataType="simple"
            data="${deltaData.values().asList()}"
            />
      </g:if>
 
      <br/> <br/>
 
      <g:if test="${lnData != null}">
           <g:barChart type="bvs"
                title="Histogram ${params.select1} (Log ${params.fieldVar}) "
            size="${[900,200]}" 
            colors="${['00FF00']}" 
            fill="${'bg,s,efefef' }"
            axes="x,y"
            axesLabels="${
               [0:lnData.keySet(),
               1:[0,lnData.values().max()/2, lnData.values().max()]
               ]
                }"
            dataType="simple"
            data="${lnData.values().asList()}"
            />
      </g:if>
 
      <br/><br/>
 
      <g:if test="${deltaLnData != null}">
           <g:barChart type="bvs"
                title="Histogram ${params.select1} (Log ${params.fieldVar} - Log previous day ${params.fieldVar}) "
            size="${[900,200]}" 
            colors="${['00FF00']}" 
            fill="${'bg,s,efefef' }"
            axes="x,y"
            axesLabels="${
               [0:deltaLnData.keySet(),
               1:[0,deltaLnData.values().max()/2, deltaLnData.values().max()]
               ]
                }"
            dataType="simple"
            data="${deltaLnData.values().asList()}"
            />
      </g:if>
    </body>
</html>

The pairs view

The pairs view is almost a copy of the main view. The difference being that in this view you can specify an extra symbol.

Result

The result is displayed in the table below for GLD, SPY, DIA and their respective differences. The data comes from historical data of end of day OHLC prices and volume.

Open High Low Close Volume
DIA DIA Open DIA High DIA Low DIA Close DIA Volume
SPY SPY Open SPY High SPY Low SPY Close SPY Volume
GLD GLD Open GLD High GLD Low GLD Close GLD Volume
DIA – SPY DIA-SPY Open DIA-SPY High DIA-SPY Low DIA-SPY Close DIA-SPY Volume
DIA – GLD DIA-GLD Open DIA-GLD High DIA-GLD Low DIA-GLD Close

DIA-GLD Volume
SPY – GLD SPY-GLD Open SPY-GLD High SPY-GLD Low SPY-GLD Close

SPY-GLD Volume

Conclusions

Turns out that the distributions are pretty non random, almost Gaussian. So we may be able to predict stock prices, a little a bit more accurately than you would have expected.

  • Share/Bookmark

August 29, 2010

Grails Finance Services

Filed under: programming — Tags: , , — admin @ 8:16 pm

Grails Finance 0.7

I should have started with Grails services from the beginning. Services are just another layer in the architecture. Services can easily be injected, wherever you need them. I also ran into some trouble, because of upgrading to a new Grails version.

Creating services

So I did a Mac Ports upgrade, which included upgrade of Grails to 1.3.4. When I tried to create a new service with

1

grails create-service simulation

Grails complained about a version mismatch, so I had to do

1

grails update

to fix that. The service

1
2
3
4
5
6
7
8
9
10
11

class SimulationService {
 
    static transactional = true
 
    def create(name, description) {
	def simulation = new Simulation(name, description)
	simulation.save()
 
	return Simulation.get(simulation.id)
    }
}

can now be injected into the controller for example.

1
2
3
4
5
6
7
8

class SimulationController {
   def scaffold = true
	def simulationService
 
	def persist(name, description)  {
		def simulation = simulationService.save(name, description)
	}
}

Testing services

Grails created a unit test stub along with the service, but I also wanted an integration test.

1

grails create-integration-test SimulationServiceIntegration

Gant then suddenly reported a MissingPropertyException. Very mysterious. It turns out this is a known bug/issue GRAILS-6606. The fix was easy – adding the line

1

def type = "Tests"

to CreateIntegrationTest.groovy

1
2
3
4
5
6
7
8
9
10
11

...
target ('default': "Creates a new Grails integration test which loads the whole Grails environment when run") {
    depends(checkVersion, parseArguments)
 
    def type = "Tests"
    promptForName(type: "Integration test")
 
    def name = argsMap["params"][0]
    name = purgeRedundantArtifactSuffix(name, type)
    createIntegrationTest(name: name, suffix: "")
}

The bad news is, that there are bugs/issues in Grails 1.3.4. The good news is that, some of these are easy to fix. Here is an example test

1
2
3
4
5
6

   ...
   def simulationService
 
    void testCreate() {
	assertNotNull simulationService
    }

Dependency Injection rules.

You can run all the tests with

1

grails test-app

Side topic: Levy distribution

It is a well known fact that stock prices etc. do not conform to the normal distribution. Apparently some people have the idea that the Levy distribution is more appropriate. One of these people is Mandelbrot of the fractals. Mandelbrot recognizes four schools of thought

  1. The most popular, technical analysts school exploiting short term changes.
  2. Random walk, statistically independent prices, but price changes are Gauss distributed.
  3. Another random walk believers school, but price changes have infinite variance.
  4. Random walk is a first order approximation, but there is also a second order!

This blog explains why the Levy distribution makes trend following profitable. Let the profits run and cut your losses. If you do it correctly, the fat tail of the distribution will make you a net profit.

Random links of interest

  • Share/Bookmark
Older Posts »

Powered by WordPress