App Design Issues

Speed-reading RAW values
Adding fields to an existing GaterBase
Iterating through large files
DataGater for data-collection apps
DataGater for data-analysis apps
Building brand new apps
Synchronizing local data repositories
Minimizing gating-fees during development
Keeping a purchase log on the client systems

Speed-Reading RAW values

Problem: Need quicker access to RAW data in Governed cases during data collection

Solution: Indexers or Cached Cases

The governor’s time and frequency settings were a compromise, aiming at a balance between data entry convenience and the need to to require payment.

The governor is essential to the ability to generate revenue. Payment achieves fast-access to multiple cases. So disabling the governor altogether or hacking a way around it will, in most cases, undercut your revenue stream.

But every app has its own unique needs and demands. For example, if your users are going to want to review, sort or cull records from a larger number of records ( prior to any purchasing) there are 2 approaches to consider

1. Use Indexer fields for the variables they are likely to be reviewing or sorting by. This is a good solution if there are only a couple of such fields – but the GaterBase does not allow more than 3 indexer variables – and you will probably need at least one for the key Case identifier field.

2. Do Case caching – in live memory. If the user is entering data into a lot of cases in a hurry, and wants to scroll back over the data on recently-entered cases, if data access was directly from the GaterBase, they would be likely to trigger a speed bump. But if you kept some number of recently-entered cases in a cache (such as a limited-size array of Case objects) you could access recent case data without hitting the GaterBase … and avoid the governor all together.

For example,
           Dim oCachedCases(15) as DataGater.Repository.Case
    Or
           Dim colCachedCases as DataGater.Repository.Case

And use a procedure to limit the number of cases in the cache to some number (like 15) by removing the lowest-indexed CachedCase when a 16^th one is added.

You should avoid saving cached cases to disk, especially if data collection is a significant function of your app, because an enterprising hacker might grab the data off the disk and bypass the purchase transaction through which you get paid.

Adding New FieldGroups and Fields to a GaterBase

If the GaterBase already has some stored data

In surveys, questionnaires, interviews and many other kinds of research, good practice dictates that if you change a data collection instrument partway into a study, you should start all over again: empty out all the data collected so far and begin again.

But there are circumstances where that rule does not apply. For example, you might have conducted a pre-test interview on a group of people, but did some reworking of a call-back interview’s questions before the call-backs were begun.

If no data has been acquired
There is no problem. GaterBase has the methods to do it

To add a new Field
        Get the FieldGroup object that you will be adding to

                  oFieldGroup(GroupIndex) = oCaseDefinition _
                      .FieldGroups.GetAt(GroupIndex)

Create the new FieldDefinition
Set oFieldDefinition = New _
FieldGroupDefinitionFactory.Create

Add it to the FieldGroup
oFieldGroup(GroupIndex).Add oFieldDefinition

          Set the modified FieldGroup into the CaseDefinition object
                      oCaseDefinition.FieldGroups _
                    .SetAt(GroupIndex, oFieldGroup)

Place the modified CaseDefinition into the GaterBase (making sure it is not open when you do so)
If oGaterBase.IsOpen Then oGaterBase.Flush
Set oGaterBase.CaseDefinition = oCaseDefinition

To add a new FieldGroup

Create a new FieldGroup object
    Set oFieldGroup(GroupIndex) = New _
         oFieldGroupFactory.CreateWithName _
        (sFieldGroupNames(GroupIndex)

Set its component FieldDefinitions into it
oFieldGroup(GroupIndex).Add oFieldDefinition

and add it to the CaseDefinition
oCaseDefinition.FieldGroups.Add oFieldGroup

Place the modified CaseDefinition into the GaterBase (making sure it is not open when you do so)

If oGaterBase.IsOpen Then oGaterBase.Flush
Set oGaterBase.CaseDefinition = oCaseDefinition

Speeding up case-iterations with large data files

Problem: Listing and sorting. If there is a large number of cases in a file, it can take a while to do operations that require iterating through them: listing all the names on file, for example, or doing a search or a sort.
Opening a case from the GaterBase takes very little time but
doing it for thousands of cases can add up to a perceptible delay

Solution 1: Give the user a message and a progress screen. Notify the user about what is going on, and to expect a delay.

An hour-glass mouse-pointer is not sufficient – particularly when is just sits there for a minute or more. Users can get really anxious. Keeping them informed goes a long way towards damping confusion and soothing frustration

You might want to use a progress bar –whose value reflects the number of cases read so far. Set its max property to the number of cases on file
ProgressBar.Max = oGaterBase.NumberOfCases

Solution 2: Use a Cache of Ungoverned Cases..

This is a good way to access values of all the Indexer fields (like ID’s and names) in a hurry.

Filling the array or collection the first time will take a little while (be sure to alert the user) but from then on, the program will be able to move a lot faster through them. The Cached Cases will contain all the indexer field values (like record numbers and case names).
Dim oUngovernedCases( ) as DGCase

And for each Case on File, open an ungoverned one and put it in an array or collection:
oUngovernedCases(iCaseID) = oGaterBase _
.OpenCaseUngovernedByID(iCaseID)

Be sure to add, remove or edit cached cases every time a Case on file is added, removed or edited.

Adapting Data Collection Apps to GaterWare

Problem: Adapting a data-collection app that currently stores data in a standard (public) file format to a GaterWare billing system.

If your app’s value to the user is the collection of data, and the app currently stores data in a publicly readable format (like Excel, XML, Access or .CSV ASCII tables), then users could get to the data without a payment transaction.

Converting the app to GaterWare requires that the data be saved to a GaterBase repository first, as RAW data. The user then makes a purchase transaction that enables it for export and analysis. And you, the app vendor, will get paid.

Solution 1: Full Replacement. Replace all data input/output properties and methods with DataGater ones.

This will end up having the simplest and most serviceable code. But it may require more development and programming time to achieve.

Solution 2: Substitute Class. Enngineer a GaterBaseDataManager Class, with calls like the ones of your current repository, and swap in for the original data manager.

This will be possible if the old system’s data repository is managed as a class that exposes sufficiently similar methods and properties.

The first time this is implemented, it will probably take more development and programming time than the other solutions, but having a re-usable component that can just be swapped in with a global search and replace, or with a name-change, would be very useful if you were planning to convert a number of apps in the same way.

Solution 3: Active-Memory Buffer. Put data into active memory, using the old architecture, and then import it into a GaterBase before anything is saved to disk. This can work only if the old system holds data in memory before committing to disk. The advantage is that you may be able to get away with having to write less new code than with the other solutions.

Adapting Data Processing Apps to GaterWare

Problem: Distributing a data-processing app as GaterWare.

If a processing app (like an analytics or reporting program) works on publicly readable data (like Excel, XML, Access or .CSV ASCII tables), then users will be able to do their processing without going through the DataGater service. A way is needed to require users to do data-gating before analysis begins.

Solution 1: Import and Convert.

The developer adds an “Import Data” routine to the app, adds a “Payment Transactions” page, and converts all the data I/O calls to GaterBase. This way, the user gets the original, publicly-readable data, reads it into the app, pays a gating fee, and does the processing

If the app is being written from the ground up, this is probably the best way do go.

Solution 2: Import and Verify

The developer adds an “Import Data” routine to the app, adds a “Payment Transactions” page, and adds a “VerifyPuchase” routine to prevent analysis from using any values in the original datafield that do not match the GaterBase PAID values. The actual processing would be done on the original data with the original code.

Development time would be substantially reduced with this approach.

The verification routine would have to be engineered to make bypassing it very hard to do. If there is a substantial risk of crackers being able to disable or otherwise cheat the verification procedures and thereby avoid payment, this might be a solution to avoid.

Building Brand New Apps

Problem: How to design a brand new processing app as GaterWare

Solution 1: Full Model. Use the GaterBase for all data management.

Analyze directly from GaterBase, import RAW data into it, the user makes payment and does all processing with PAID data.

Pro: Simple design, few steps, no need for payment-verification
Con: Not as simple to apply SQL queries, sorts etc.

Solution 2: Verification but Use Original Model. Use the GaterBase as a billing and payment-verification tool. Analyze from original file

Import RAW data into it; make payment; do processing on the original data file but verify that its matches the GaterBase PAID data.

Pro: Can use familiar data analysis tools and routines adapted to the original data file format.
Con: Need for payment verification, risk of cracking by cheating clients

Solution 3: Verification and Export Model. Use the GaterBase as a billing and payment-verification tool, analyze from an exported file.

Import RAW data to it; make payment; export PAID data to a new format file; do processing on the new data file while verifying match to GaterBase data.

Pro: Can use familiar data analysis tools and routines adapted to the exported data file format.
Con: Need for payment verification, extra step to export data, risk of cracking by cheating clients

Synchronizing multiple data repositories

The GaterBase was designed to make it easy to set up multiple data repositories on local machines, and to synchronize the data across them.

Creating multiple data repositories.

The original GaterBase repository is called “Home”. Others are “Remotes”
Once the Home GaterBase has been created, Remotes are generated from it with a single method call.

Dim oRemoteGaterBase as DGGaterBase
oRemoteGaterBase = oGaterBase _
.CreateRemoteGaterBase (CurrentCaseIDArray(), _
Filename, AssignableLowID,AssignableHighID)

The iCaseIDArray( ) passes the HomeGaterBase records into the Remote. The AssignableLow and High ID’s are the CaseID’s that the new repository will allow to be created locally.

About setting assignable ranges

Ranges are set to ensure that records do not get overwritten when different local repositories get synchronized.

You will not be able to create a record on a local repository outside the assigned range.

The ranges do not affect the actual size of the GaterBase files.

Set a range that is several times the range you actually expect to need: If you expect to create 100 -200 new records in each of 3 local GaterBases, then you might set the ranges as follows

HomeGaterBase:           1          to         1000
RemoteA                      1001     to         2000
RemoteB                      2001     to         3000

Remember that a Case’s CaseID number is an index used and managed by the GaterBase. You cannot change it once a case has been created. And if you have multiple data-entry computers with local repositories, the computers’ CaseID numbering might not seem very logical to end users. So you may decide to create a different index for them to keep track of their Cases.

I often create a “Case number” index to serve this purpose, saving it as an integer value under the field name “CaseNum” and setting the field as an indexer to facilitate quick indexing, searching and listing.

Synchronizing repositories

Local GaterBase repositories can be synchronized with a single method call from a Home GateBase, referencing a Remote.

The procedure needs the two GaterBase objects to be synchronized and an object to receive the success report. The two GaterBases, of course, must be using the same CaseDefinition structure.

Synchronizing is a one-way process. Data are added from Remotes to the Home file – but not from Home to the Remotes.

Dimension a report object and a factory

Dim colSynchIssueCollection as _
DGSynchronizationAdjudicationIssueCollection
Dim oSynchIssueCollectionFactory as _
DGSynchronizationAdjudicationIssueFactory

Instantiate new ones

DimSet oSynchIssueCollectionFactory = New _
DGSynchronizationAdjudicationIssueCollectionFactory

Set colSynchIssuesCollection = _
oSynchIssueCollectionFactory.Create

Call the Synchronize method from one of the GaterBases

colSynchIssueCollection = oGaterBase _
.Synchronize(otherGaterBase)

If the operation is completely successful, the issue count in the report will be zero
colSynchIssues.Count = 0

If one or more adjudication issues were encountered, their descriptions can be accessed through the Issue objects contained in the collection

Dimension and Instantiate the factory and a new Issue object

                    Dim oSynchIssue as _
                DGSynchronizationAdjudicationIssue
                    Dim oSynchIssueFactory as _
                DGSynchronizationAdjudicationIssueFactory
           Set oSynchIssueFactory = New _
                DGSynchronizationAdjudicationIssueFactory
           Set oSynchIssue = oSynchIssueFactory.Create
           oSychronizationAdjudicationIssue = _
                colSychIssueCollection.GetAt(i)

Read. For each issue in the collection, read its properties and decide what to do, if anything (it may have been resolved)

.CaseID                               the case it happened with
.ConflictReason                description of the conflict
.ConfllictResolution    description of the resolution

In the event that 2 values for the same field differ, the synchronizing function is set to resolve conflicts in favor of value most-recently set. And a defined value takes precedence over an undefined (missing) value.

Minimizing Data Purchase during development

Problem: Analysis tools require PAID data sets.
This could add to the cost of developing them. There are ways of minimizing or even eliminating the gating charges.

Solution 1: (Free) Use Indexer Fields.
There can be up to 3 of Indexer fields in a GaterBase. If your analysis tools work with strings or integer fields, then temporarily set up a “test bench” data set with the data to be analyzed defined as indexers.

1.       Define the to-be-analyzed fields as Indexers
2.       Enter/acquire the data
3.       Do NOT pay for gating .. or, if you need more fields, beyond the indexers, restrict the fields to be purchased to only the critical ones.
4.       Open cases as ungoverned (which is required for analysis operations).They will contain all the data in the free indexer fields, and any other data which you had chosen to pay for.

Note that you cannot change a previously-declared indexer field into a non-indexer. So be sure to set up the case definition for your test-bench data with the indexer fields you will be using right from the start.

Solution 2. (Free) Use a Pre-PAID data set
Select one supplied with this SDK or download one from the DataGater website. The samples will include data of all the available variable types: Sting, Integer, FloatingPoint, Boolean and Join.

Solution 3. (Contained-cost) Reduce the number of fields you pay for to practice on or test against
Purchase transactions allow you to specify which fields and cases you want to get gating-keys for.

Restrict your request (and payment) to just those fields that are critical to the analysis you are engineering, and restrict your number of records to the minimum, too.

Keep a local purchase event log

There are two reasons you might decide to keep a log-file of purchase events on the client’s machine.

For debugging
During your app’s Beta development period, a local log file can help you identify just when and where gating issues arrive in the field. At roll-out time, you may decide to disable this feature.

For adjudication
If a client submits a claim to you for repayment, asserting that the software or service failed them, you can refer to the log-file to ascertain what really happened and resolve the issue appropriately. Against that possibility, you might also include an app activity log-file of your own. You might consider encrypting them, to reduce the risk of client tampering.

Among the advantages of a micro-billing system like DataGater is that the value of an individual claim is so little that
1) customer’s are not likely to pursue claims and
2) it will cost you little to make good on a money-back guarantee.

Creating multiple data repositories.

About setting assignable ranges

Synchronizing repositories

For debugging During your app’s Beta development period, a local log file can help you identify just when and where gating issues arrive in the field. At roll-out time, you may decide to disable this feature.

For debugging
During your app’s Beta development period, a local log file can help you identify just when and where gating issues arrive in the field. At roll-out time, you may decide to disable this feature.