User:Bpeller: Difference between revisions

From KDE TechBase
 
(268 intermediate revisions by the same user not shown)
Line 1: Line 1:
= '''Central Stock Price Information'''  =
= '''Central Stock Price Information'''  =
''A proposal for Season of KDE 2010 by Brian Cappello''<br>
''Mentors: Cristian Onet and Alvaro Soliverez


'''Central/Master User Interface:'''
== Use Cases ==
:*Provide a widget/plasmoid/application that allows for:
:*A plasmoid wants to display some stock quotes stored in a watchlist.
:::Viewing/browsing watchlists.
:*Santosh Kumar wants to provide notification support for changes in stock prices.
:::Managing symbols and watchlists.
:*An investor wants to chart an investment's yearly performance starting from their first purchase thirty-five years ago.
:::Configuring the backend's default data source and resource/performance settings.
:*An investor wants to do some fundamental research on a stock they're considering trading.
:*A day trader wants to chart today's 5-minute price action since the open.
:*A swing trader wants to chart two months worth of hourly data.
:*Another swing trader wants to chart 200 bars of daily data.
:*A student of the markets wants to "replay" the crash of '87 starting from 3 months prior and advancing one day at a time.
:*A mechanical systems trader writes a scripted trading algorithm, and they want to back-test its performance on various sets of historical data.


== DBus Interfaces for developers: ==
[http://techbase.kde.org/User:Bpeller# [Jump to Top]]


:*'''Management of stored symbols and watchlists'''
:::Clients can add/request-to-remove symbols to/from the database.
:::Clients can add/retrieve/request-to-modify global watchlists and use them locally.


:*'''Intraday Quotes / most recent close''' (if after market hours)  
== Tentative Schedule: WIP/subject to change ==
:::Input: Symbol
=== May W4: ===
:*Continue research into DBus, asynchronous transfers, doctools, unit tests and cmake.
:*Get some DBus code working!
=== June W1: ===
:*Attempt to "finalize" class/object separation and interaction planning.
:*Preliminary DBus symbol management. Add/remove only, no watchlists yet.
:*Implement Yahoo historical data interpreter and runtime-only storage. (Concentrate on daily data. W/M/IntraDayHistory later.)
=== June W2: ===
:*Implement Yahoo intraday quotes support and runtime merging/storage into most-recent-index of daily historical data.
:*Get on-demand DBus intraday support working.
:*Figure out how best to store data to HDD. Possibly as flat-file CSVs and/or KConfig?
:*Get SymbolManager storage-to/from-disk working.
=== June W3: ===
:*Get intraday quotes monitoring working. Broadcast notification of updates to clients.
:*Implement IntraDay quotes test cases.
:*Get preliminary on-demand DBus CompanyInfo/Stats working (assume client knows what keys are available).
=== June W4: ===
:*CompanyInfo Test cases.
:*Begin to get HDD saving/loading historical data working (maybe merging/updating).
=== July W1: ===
:*Finish up historical data HDD stuffs, including daily, weekly, monthly, and merging/updating.
=== July W2: ===
:*Begin work on DBus interface for historical data. No IntraDayHistory support yet.
=== July W3: ===
:*Finish DBus interface for historical data, including time conversion and adjusted prices.
:*Finish/polish Previous-Close convenience function.
=== July W4: ===
:*Implement Historical Data and Previous-Close test cases.
:*Historical Data performance profiling and optimization.
=== August W1: ===
:*Add DBus WatchList support.
:*Watchlist test cases.
:*Polish CompanyInfo/Stats DBus interface, add support for querying, key-name lookup, etc.
=== August W2: ===
:*Saving/loading of CompanyInfo/Stats to disk.
:*Improve/update CompanyInfo test cases.
=== August W3: ===
:*Polish documentation, test cases, and further profiling.
=== August W4: ===
:*Buffer


:::Output: SingleBarStruct: KDateTime, Open, High, Low, Close, Volume [and Open Interest, if applicable. ''Optional:'' Beta]
=== Time Permitting: ===
:*Expand CompanyInfo by parsing data from Yahoo's Key Statistics page.
:*Improved intraday real-time support by parsing data from finance.yahoo.com (or other source?)
:*User-controlled CSV export.
:*Add/finish support for IntraDayHistory.
:*IntraDayHistory test cases.


:*'''Previous Close''' (convenience function)
[http://techbase.kde.org/User:Bpeller# [Jump to Top]]
:::Input: Symbol


:::Output: SingleBarStruct: KDateTime, Open, High, Low, Close, Volume [and Open Interest, if applicable. ''Optional:'' Beta]
== Planned Components ==
*'''Always-On/On-Demand Backend:''' A member of [[Projects/KdeFinance/Alkimia|Alkimia]]
::''The backend handles all quote retrieval/storage, and provides a public interface for client applications to query over DBus for whatever information they need. The backend also serves as a centralized place for consolidating various stock-price-manipulation maths common to financial applications' needs.''


:*'''Company Information, Key Statistics and Fundamentals'''
*'''A "Master" Backend UI:''' Some form/combination of plasmoid/application/kpart/kcm...
:::Inputs:
::''Cumulative functionality should support something along the lines of:''
:::*Managing which symbols and watchlists the backend permanently stores data for.
:::*Configuring the backend's default historical and intra-day data sources. (And maybe resource/performance settings.)
::::::YahooCSV: Download quotes from Yahoo! ''(Default for Intra-Day & Historical)''
::::::Directory: Import/monitor quotes from a directory containing CSV files. ''(For users with paid-data services.)''
::::::''Time Permitting:'' YahooWEB: Parse real-time data from the finance.yahoo.com website. ''(Intra-Day only)
:::*Exporting user specified data/information/statistics to CSV.
 
[http://techbase.kde.org/User:Bpeller# [Jump to Top]]
 
 
== Planned Backend Functionality ==
 
=== '''Internals''' ===
:*Maintain/store a '''master list of all watchlists.'''
:*Maintain/store a '''master list of all symbols''' to be watched. Filters out watchlist-duplicates and includes symbols which aren't a member of any watchlist.
:*Maintain a '''local/offline store of downloaded data:'''
:#Company info and statistics. ''(Used for offline mode/throughout the day. Company stats change frequently enough and are reasonably cheap (fast) to download so we update this once daily, if possible.)''
:#Accumulate sampled Intra-Day data as Minutely data.
:#Store and keep up-to-date "primary" data: Minutely, Daily, Weekly, and Monthly Time Periods
 
[http://techbase.kde.org/User:Bpeller# [Jump to Top]]
 
 
=== '''Public DBus Interfaces''' ===
==== '''Management of stored symbols and watchlists''' ====
''These are primary functions.''
===== '''Symbol Management''' =====
:*''Add/Remove Symbols''
:::'''Inputs:'''
::::*Symbol
::::*Action: Add or Remove
:::'''Output:'''
::::*Bool Success
 
:*''Modify a watchlist's symbols''
:::'''Inputs:'''
::::*Symbol
::::*Watchlist
::::*Action: Add or Remove
:::'''Output:'''
::::*Bool Success
 
===== '''Watchlist Management''' =====
:*''Get Watchlists''
:::'''Inputs:'''
:::'''Output:'''
::::*List of all watchlist names
 
:*''Get Watchlist Symbols''
:::'''Inputs:'''
::::*Watchlist name
:::'''Output:'''
::::*List of the watchlist's symbols
 
:*''Add/Remove Watchlists''
:::'''Inputs:'''
::::*Watchlist
::::*Action: Add or Remove
:::'''Output:'''
::::*Bool Success
 
:*''Lower Priority: Merge Watchlists''
:::'''Inputs:'''
::::*First Watchlist
::::*Second Watchlist
::::*Action: Merge or New
::::::Merge: First into second, delete first. [Mirroring 'mv' command.]
::::::New: Create a new watchlist, preserve originals.
::::::*''If Reqd:'' New Watchlist's Name
:::'''Output:'''
::::*Bool Success
 
[http://techbase.kde.org/User:Bpeller# [Jump to Top]]
 
 
==== '''Intra-Day/Most-Recent-Close Quote''' ====
:''This is a convenience function.''
:::'''Inputs:'''
::::*Symbol
::::*''Optional'': Prices or Percents
::::::Prices: the real prices ''(Default)''
::::::Percents: Percent changes from previous close
 
:::'''Output:'''
::::*struct: Contains a KDateTime, the Open, High, Low, Close, Volume, and, if applicable, Open Interest (eg for futures). ''Optional'': and Beta.
 
==== '''Previous Day's Close Quote''' ====
:''This is a convenience function.''
:::'''Inputs:'''
::::*Symbol
::::*''Optional'': Prices or Percents
::::::Prices: the real prices ''(Default)''
::::::Percents: Percent changes from previous close (eg two closes ago)
 
:::'''Output:'''
::::*struct: Contains a KDateTime, the Open, High, Low, Close, Volume, and, if applicable, Open Interest (eg for futures). ''Optional'': and Beta.
 
===='''Company Information, Key Statistics and Fundamentals''' ====
:''This is a primary function.''
:::'''Inputs:'''
::::*Symbol
::::*Symbol
::::*Parameters (keys)
::::*Parameters (keys)
::::::''Ideally, there should probably be a way to query what keys are available, as well as key-lookup by human-readable names.''
::::::''Initially most (all applicable) parameters [http://www.etraderzone.com/free-scripts/50-yahoo-stock-quotes.html easily downloadable] from Yahoo will be supported. Time permitting, more information sources can be added (such as parsing parameter values from Yahoo's [http://finance.yahoo.com/q/ks?s=NOK+Key+Statistics Key Statistics] webpage).
 
::::::''Time Permitting: There should probably be a way to query at runtime what keys are available, as well as key-lookup by human-readable names.''


:::Outputs:
:::'''Output:''' ''Must be able to accommodate both strings and numbers.''
::::*A container object holding the requested parameter values.
::::*A container object holding the requested parameters' key-value pairs.


:*'''Historical Quotes'''  
[http://techbase.kde.org/User:Bpeller# [Jump to Top]]
:::Inputs:
 
 
==== '''Historical Quotes''' ====
:''This is a primary function.''
:::'''Inputs:'''
::::*Symbol  
::::*Symbol  
::::*Time Period: Specify by providing:
::::*Amount/duration of historical data:
:::::::A major time unit. (Minutely, Daily, Weekly, Monthly)
:::::::''Optional:'' Specify an end date. ''(Defaults to the most recent close.)''
:::::::''Optional:'' A multiplier. Default = 1.
::::*Length of historical data:
:::::::''Optional:'' Specify an end date. Defaults to the most recent close.
:::::::Specify a duration (relative to the end date) by providing one of the following:
:::::::Specify a duration (relative to the end date) by providing one of the following:
:::::::::A start date.
:::::::::*A start date.
:::::::::A length of elapsed time units, independent of the requested time period.
:::::::::*An elapsed quantity and unit of time, independent of the (optionally) requested time period.
:::::::::Or directly as a number of bars.
:::::::::*Or directly as a number of bars.
::::*''Optional:'' Type of historical data:
::::*''Optional:'' Time Period: ''Specify by providing'':
:::::::Adjusted: (Default) Prices adjusted for splits and dividends.
:::::::''Optional:'' A major time unit. (Daily ''(Default)'', Weekly, or Monthly)
:::::::Absolute: Raw price data. (Provided by default if the stock has no splits/dividends.)
:::::::''Optional:'' A multiplier. ''Default = 1''
::::*''Optional:'' A maximum lookback. Default = 0.
::::*''Optional:'' Adjusted or Absolute Prices
:::::::Adjusted: ''(Default)'' Prices adjusted for splits and dividends.
:::::::Absolute: Raw price data. ''(Provided by default if the requested symbol has no splits or dividends.)''
::::*''Optional:'' A maximum lookback. ''Default = 0''
:::::::''The number of "hidden" bars (older than the start date) to return. Used by technical analysis indicators so their results will be complete for the requested length of "shown" price data.''
:::::::''The number of "hidden" bars (older than the start date) to return. Used by technical analysis indicators so their results will be complete for the requested length of "shown" price data.''
:::Outputs:
 
::::*A container object holding the input parameters used to produce the data, as well as the data itself, stored as a list of SingleBarStructs.
:::'''Output:'''
::::*A container object holding the input parameters used to produce the data, as well as the data itself (containing the same information as Intra-Day structs). ''Need to look into what would be the most convenient storage method for MVC-based clients such as KDChart as used by KMyMoney, but that's still flexible enough to be non-MVC-friendly.''
 
[http://techbase.kde.org/User:Bpeller# [Jump to Top]]




'''Dreams for the Future''' (hopefully reasonable)
== Future Backend Possibilities ==
#Support for calculating relative price/performance between multiple data sets.
#Support for calculating custom indicies from watchlists.
#Store symbols and watchlists using Nepomuk relationships. Symbols can be associated with a Sector, an Industry, and Market Capitalization.
#Support for browsing/querying sectors and industries.
#Technical Analysis Indicators: [Maybe] Essentially this would be providing a Qt-like API wrapped around TA-Lib's [http://ta-lib.svn.sourceforge.net/viewvc/ta-lib/trunk/ta-lib/c/include/ta_abstract.h?view=markup Abstract interface]. TA-Lib [http://ta-lib.org/d_api/d_api.html#Technical%20Analysis%20Functions expects] data to be stored in std c++ arrays, as well as requiring the constant reuse of calculated source data, so from a convenience/cache-efficiency/multithreading standpoint it probably makes the most sense to centralize indicator support inside Alkimia. ''However, TA-Lib isn't exactly available for most (all?) distributions, let alone included by default, so that's definitely something to keep in mind. [Then again, aside from [http://qtstalker.sourceforge.net/ Qtstalker], I don't know of any even remotely-active *Linux* projects using it so... what incentive do they have?]''


''Time Permitting / Low Priority:''


#Support for calculating relative price/performance between multiple data sets.
#Support for calculating "custom Market Indexes" from watchlists.
#Support for storing/browsing/querying sectors and industries, as well as tracking their performance using (1) and (2).
#Improved support for options, futures, and FOREX.
#[Maybe] Technical Analysis Indicator Support: Essentially this would be providing a Qt-like API wrapped around [http://ta-lib.org/index.html TA-Lib]'s [http://ta-lib.org/d_api/d_api.html#Abstraction Abstract Interface]. Interface documentation [http://ta-lib.svn.sourceforge.net/viewvc/ta-lib/trunk/ta-lib/c/include/ta_abstract.h?view=markup here]. ''If this were to be done, it would be as an optional plugin, detected at runtime.''
#[Depends on TA-Lib] Manual calculation of Beta with controllable inputs for the time period and the "volatility base market." ''This ability is probably most applicable for large private investors, portfolio analysts, and institutions/banks, and is particularly relevant for those invested in international/emerging markets who do not want to base their volatility calculations off of American markets, as Yahoo's provided Beta [http://www.investopedia.com/articles/financial-theory/09/calculating-beta.asp is assumed] to do.''


'''Crazy Day Dreaming'''
[http://techbase.kde.org/User:Bpeller# [Jump to Top]]
#Support for uploading/merging historical data stuffs to some form of P2P/cloud. Initially used for the sharing of non-freely (or at least easily) downloadable data such as minutely data, stock split dates/ratios, and daily/weekly/monthly data older than what Yahoo's servers provide.

Latest revision as of 08:36, 28 May 2010

Central Stock Price Information

A proposal for Season of KDE 2010 by Brian Cappello
Mentors: Cristian Onet and Alvaro Soliverez

Use Cases

  • A plasmoid wants to display some stock quotes stored in a watchlist.
  • Santosh Kumar wants to provide notification support for changes in stock prices.
  • An investor wants to chart an investment's yearly performance starting from their first purchase thirty-five years ago.
  • An investor wants to do some fundamental research on a stock they're considering trading.
  • A day trader wants to chart today's 5-minute price action since the open.
  • A swing trader wants to chart two months worth of hourly data.
  • Another swing trader wants to chart 200 bars of daily data.
  • A student of the markets wants to "replay" the crash of '87 starting from 3 months prior and advancing one day at a time.
  • A mechanical systems trader writes a scripted trading algorithm, and they want to back-test its performance on various sets of historical data.

[Jump to Top]


Tentative Schedule: WIP/subject to change

May W4:

  • Continue research into DBus, asynchronous transfers, doctools, unit tests and cmake.
  • Get some DBus code working!

June W1:

  • Attempt to "finalize" class/object separation and interaction planning.
  • Preliminary DBus symbol management. Add/remove only, no watchlists yet.
  • Implement Yahoo historical data interpreter and runtime-only storage. (Concentrate on daily data. W/M/IntraDayHistory later.)

June W2:

  • Implement Yahoo intraday quotes support and runtime merging/storage into most-recent-index of daily historical data.
  • Get on-demand DBus intraday support working.
  • Figure out how best to store data to HDD. Possibly as flat-file CSVs and/or KConfig?
  • Get SymbolManager storage-to/from-disk working.

June W3:

  • Get intraday quotes monitoring working. Broadcast notification of updates to clients.
  • Implement IntraDay quotes test cases.
  • Get preliminary on-demand DBus CompanyInfo/Stats working (assume client knows what keys are available).

June W4:

  • CompanyInfo Test cases.
  • Begin to get HDD saving/loading historical data working (maybe merging/updating).

July W1:

  • Finish up historical data HDD stuffs, including daily, weekly, monthly, and merging/updating.

July W2:

  • Begin work on DBus interface for historical data. No IntraDayHistory support yet.

July W3:

  • Finish DBus interface for historical data, including time conversion and adjusted prices.
  • Finish/polish Previous-Close convenience function.

July W4:

  • Implement Historical Data and Previous-Close test cases.
  • Historical Data performance profiling and optimization.

August W1:

  • Add DBus WatchList support.
  • Watchlist test cases.
  • Polish CompanyInfo/Stats DBus interface, add support for querying, key-name lookup, etc.

August W2:

  • Saving/loading of CompanyInfo/Stats to disk.
  • Improve/update CompanyInfo test cases.

August W3:

  • Polish documentation, test cases, and further profiling.

August W4:

  • Buffer

Time Permitting:

  • Expand CompanyInfo by parsing data from Yahoo's Key Statistics page.
  • Improved intraday real-time support by parsing data from finance.yahoo.com (or other source?)
  • User-controlled CSV export.
  • Add/finish support for IntraDayHistory.
  • IntraDayHistory test cases.

[Jump to Top]

Planned Components

  • Always-On/On-Demand Backend: A member of Alkimia
The backend handles all quote retrieval/storage, and provides a public interface for client applications to query over DBus for whatever information they need. The backend also serves as a centralized place for consolidating various stock-price-manipulation maths common to financial applications' needs.
  • A "Master" Backend UI: Some form/combination of plasmoid/application/kpart/kcm...
Cumulative functionality should support something along the lines of:
  • Managing which symbols and watchlists the backend permanently stores data for.
  • Configuring the backend's default historical and intra-day data sources. (And maybe resource/performance settings.)
YahooCSV: Download quotes from Yahoo! (Default for Intra-Day & Historical)
Directory: Import/monitor quotes from a directory containing CSV files. (For users with paid-data services.)
Time Permitting: YahooWEB: Parse real-time data from the finance.yahoo.com website. (Intra-Day only)
  • Exporting user specified data/information/statistics to CSV.

[Jump to Top]


Planned Backend Functionality

Internals

  • Maintain/store a master list of all watchlists.
  • Maintain/store a master list of all symbols to be watched. Filters out watchlist-duplicates and includes symbols which aren't a member of any watchlist.
  • Maintain a local/offline store of downloaded data:
  1. Company info and statistics. (Used for offline mode/throughout the day. Company stats change frequently enough and are reasonably cheap (fast) to download so we update this once daily, if possible.)
  2. Accumulate sampled Intra-Day data as Minutely data.
  3. Store and keep up-to-date "primary" data: Minutely, Daily, Weekly, and Monthly Time Periods

[Jump to Top]


Public DBus Interfaces

Management of stored symbols and watchlists

These are primary functions.

Symbol Management
  • Add/Remove Symbols
Inputs:
  • Symbol
  • Action: Add or Remove
Output:
  • Bool Success
  • Modify a watchlist's symbols
Inputs:
  • Symbol
  • Watchlist
  • Action: Add or Remove
Output:
  • Bool Success
Watchlist Management
  • Get Watchlists
Inputs:
Output:
  • List of all watchlist names
  • Get Watchlist Symbols
Inputs:
  • Watchlist name
Output:
  • List of the watchlist's symbols
  • Add/Remove Watchlists
Inputs:
  • Watchlist
  • Action: Add or Remove
Output:
  • Bool Success
  • Lower Priority: Merge Watchlists
Inputs:
  • First Watchlist
  • Second Watchlist
  • Action: Merge or New
Merge: First into second, delete first. [Mirroring 'mv' command.]
New: Create a new watchlist, preserve originals.
  • If Reqd: New Watchlist's Name
Output:
  • Bool Success

[Jump to Top]


Intra-Day/Most-Recent-Close Quote

This is a convenience function.
Inputs:
  • Symbol
  • Optional: Prices or Percents
Prices: the real prices (Default)
Percents: Percent changes from previous close
Output:
  • struct: Contains a KDateTime, the Open, High, Low, Close, Volume, and, if applicable, Open Interest (eg for futures). Optional: and Beta.

Previous Day's Close Quote

This is a convenience function.
Inputs:
  • Symbol
  • Optional: Prices or Percents
Prices: the real prices (Default)
Percents: Percent changes from previous close (eg two closes ago)
Output:
  • struct: Contains a KDateTime, the Open, High, Low, Close, Volume, and, if applicable, Open Interest (eg for futures). Optional: and Beta.

Company Information, Key Statistics and Fundamentals

This is a primary function.
Inputs:
  • Symbol
  • Parameters (keys)
Initially most (all applicable) parameters easily downloadable from Yahoo will be supported. Time permitting, more information sources can be added (such as parsing parameter values from Yahoo's Key Statistics webpage).
Time Permitting: There should probably be a way to query at runtime what keys are available, as well as key-lookup by human-readable names.
Output: Must be able to accommodate both strings and numbers.
  • A container object holding the requested parameters' key-value pairs.

[Jump to Top]


Historical Quotes

This is a primary function.
Inputs:
  • Symbol
  • Amount/duration of historical data:
Optional: Specify an end date. (Defaults to the most recent close.)
Specify a duration (relative to the end date) by providing one of the following:
  • A start date.
  • An elapsed quantity and unit of time, independent of the (optionally) requested time period.
  • Or directly as a number of bars.
  • Optional: Time Period: Specify by providing:
Optional: A major time unit. (Daily (Default), Weekly, or Monthly)
Optional: A multiplier. Default = 1
  • Optional: Adjusted or Absolute Prices
Adjusted: (Default) Prices adjusted for splits and dividends.
Absolute: Raw price data. (Provided by default if the requested symbol has no splits or dividends.)
  • Optional: A maximum lookback. Default = 0
The number of "hidden" bars (older than the start date) to return. Used by technical analysis indicators so their results will be complete for the requested length of "shown" price data.
Output:
  • A container object holding the input parameters used to produce the data, as well as the data itself (containing the same information as Intra-Day structs). Need to look into what would be the most convenient storage method for MVC-based clients such as KDChart as used by KMyMoney, but that's still flexible enough to be non-MVC-friendly.

[Jump to Top]


Future Backend Possibilities

Time Permitting / Low Priority:

  1. Support for calculating relative price/performance between multiple data sets.
  2. Support for calculating "custom Market Indexes" from watchlists.
  3. Support for storing/browsing/querying sectors and industries, as well as tracking their performance using (1) and (2).
  4. Improved support for options, futures, and FOREX.
  5. [Maybe] Technical Analysis Indicator Support: Essentially this would be providing a Qt-like API wrapped around TA-Lib's Abstract Interface. Interface documentation here. If this were to be done, it would be as an optional plugin, detected at runtime.
  6. [Depends on TA-Lib] Manual calculation of Beta with controllable inputs for the time period and the "volatility base market." This ability is probably most applicable for large private investors, portfolio analysts, and institutions/banks, and is particularly relevant for those invested in international/emerging markets who do not want to base their volatility calculations off of American markets, as Yahoo's provided Beta is assumed to do.

[Jump to Top]