Quantcast
Channel: ESENT Managed Interface
Viewing all 496 articles
Browse latest View live

New Post: Test driving Microsoft.Database.Isam

$
0
0
Forget the last remark cause I think it is wrong the Cursor.Move(int rows). Shouldn't this part
                // setup our index range
                if (rows < 0)
                {
                    this.SetUpperLimit();
                }

                if (rows > 0)
                {
                    this.SetLowerLimit();
                }
be the opposite? When moving forward setting the upper limit and vice versa?

New Post: Please describe various nuget packages?

$
0
0
  1. Yes, you are correct.
  2. As you have probably figured out, ManagedEsent is just a thin wrapper over the C APIs in esent.dll. The API was designed in the 1990s, and is fairly cumbersome. (It takes so many API calls just to open a database and open a table!) 'Isam' is our attempt to make it much easier to program.
  3. Yes, PersistentDictionary is contained in this nuget package. We had originally included esent.collections.dll in ManagedEsent, but as of 1.9.1.0, we split it out in to a different package, We did it for two reasons:
    -To make it more discoverable.
    -To make the packages more fine-grained.
Hope that helps!

-martin

New Post: Test driving Microsoft.Database.Isam

$
0
0
Hello Martin

Great Work!!!

Regarding Arkej's initial problem, i have the same problem on Windows 7 (64Bit).

Furthermore i found out that maybe there seems to be a marshalling problem. The JetGetColumnInfoW with the NATIVE_COLUMNBASE_WIDE structure returns szBaseTableName as Ansi (which is treated as Unicode) and szBaseColumnName is an empty string.

Therfore at ...
cursor.EditRecord["col1"] = 1;//here it goes key not found
... an EsentRecordNotFoundException("The key was not found") is raised. Furthermore this problem only exists with Columns which are "user" created because JetGetColumnInfoW seems to work correctly whith system Tables. I also verified the correct name of the columns with ESEDatabaseView (http://www.nirsoft.net) in the database, so i think JetAddColumn() API works correctly.

In the meentime I changed the source of EsentInterop/JetApi.cs ...
public int JetGetColumnInfo(
                JET_SESID sesid,
                JET_DBID dbid,
                string tablename,
                JET_COLUMNID columnid,
                out JET_COLUMNBASE columnbase)
.. to always use the Ansi NativeMethods.JetGetColumnInfo() call, which seems to work pretty for me.

Maybe this information gives you a hint.

Kind regards
Tom

New Post: Test driving Microsoft.Database.Isam

$
0
0
Tom, thanks. Another fix is to change jetapi.cs:
        // Technically, this should have worked in Vista. But there was a bug, and
        // it was fixed after Windows 7.
        if (this.Capabilities.SupportsWindows8Features)
I'll be making this change.

Arkej, I haven't had a chance to investigate the FindRecords() change yet. (I've just bought a house, and that has eaten up all my spare time. :)

-martin

New Post: Test driving Microsoft.Database.Isam

$
0
0
Thank you, Martin!
It seems you have the insights ... ;-)

Meanwhile I have investigated another problem. There seems to be a bug in the case of empty strings ("" e. g. string.Empty) and empty byte arrays (byte[]).
Microsoft.Database.Isam does not really distinguish empty strings/byte arrays and null values. If you store such empty values you get back DBNull.Value. Furthermore if you have defined ColumnFlags.NonNull, you get an EsentNullInvalidException("Null not valid").
As the Microsoft.Isam.Esent.Interop.SetColumnHelpers.SetColumn overloads do in the case of string and byte[] datatypes Microsoft.Database.Isam.ColumnAccessor should implement a similar handling.
I have modified the ColumnAccessor. SetColumn() method the following way …
...
byte[] bytes = Converter.BytesFromObject(coltyp, isAscii, obj);
int bytesLength = bytes == null ? 0 : bytes.Length;

//+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
// in the case of zero length strings and byte[] arrays set SetColumnGrbit.ZeroLength
//+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
if (0 == bytesLength && ((coltyp == JET_coltyp.Text || coltyp == JET_coltyp.Binary
        || coltyp == JET_coltyp.LongText || coltyp == JET_coltyp.LongBinary)))
{
        grbitSet = grbitSet | SetColumnGrbit.ZeroLength;
}
//+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Api.JetSetColumn(this.isamSession.Sesid, this.tableid, columnid, bytes, bytesLength, grbitSet, setinfo);

this.updateID++;
...
... which solves the problem.

New Post: Test driving Microsoft.Database.Isam

$
0
0
I have a probrem on FindRecords(), too.
Is there any walkarounds for temporary?

New Post: Test driving Microsoft.Database.Isam

$
0
0
Hallo

1. FindRecords()
I also have the FindRecords() Problem and as Arkej found out, it could be fixed by changing ...
                // setup our index range
                if (rows < 0)
                {
                    this.SetUpperLimit();
                }

                if (rows > 0)
                {
                    this.SetLowerLimit();
                }
... to ...
                // setup our index range
                if (rows < 0)
                {
                    this.SetLowerLimit();
                }

                if (rows > 0)
                {
                    this.SetUpperLimit();
                }
... of Move(int rows) function in Cursor.cs.

2. EditRecord Access after FindRecords()
Furthermore there is a minor Problem that it is possible to access the EditRecord, allthough the position of the cursor is out of range after a FindRecord() call. To fix the Problem I have to make some change here and there.

At first i added a private bool inserting field (like the exisiting updating, which keeps track if the cursor is in insert or update mode) which I only set at BeginEditForInsert() to know if the cursor is in insert mode and which is set to false, nearly everywhere where updating is set to false except in Delete() because the "updating = false" there is unnecessary, I think.
Than I changed Cursor.Checkrecord() to ...
        /// <summary>
        /// Checks the record.
        /// </summary>
        /// <param name="grbit">The optional grbit.</param>
        /// <exception cref="EsentNoCurrentRecordException">
        /// Thrown when the cursor is not on a record if not an insert of EditRecord.
        /// </exception>
        internal void CheckRecord(RetrieveColumnGrbit grbit = RetrieveColumnGrbit.None)
        {
            lock (this.isamSession)
            {
                if ((grbit & RetrieveColumnGrbit.RetrieveCopy) == 0 || ((grbit & RetrieveColumnGrbit.RetrieveCopy) != 0 & !inserting))
                {
                    if (this.outOfRange)
                    {
                        throw new EsentNoCurrentRecordException();
                    }
                }
            }
        }
At last I had to do the following changes in ColumnAccessor:

SizeOf(Columnid columnid, int index):
and
RetrieveColumn(JET_COLUMNID columnid, JET_coltyp coltyp, bool isAscii, int index)
                //if ((this.grbit & RetrieveColumnGrbit.RetrieveCopy) == 0)
                //{
                //    this.cursor.CheckRecord();
                //}
                this.cursor.CheckRecord(this.grbit);
SetColumn(JET_COLUMNID columnid, JET_coltyp coltyp, bool isAscii, int index, object obj)
                this.cursor.CheckRecord(this.grbit);
                if ((this.grbit & RetrieveColumnGrbit.RetrieveCopy) == 0)
                {
                    //this.cursor.CheckRecord();
                    throw new InvalidOperationException("You may only update fields through Cursor.EditRecord.");
                }
3. BeginEditForUpdate after FindRecords()
For consistency there should also be a CheckRecord() call in Cursor.BeginEditForUpdate() as in GetFields(), Position and Location. For now it is possible to issue BeginEditForUpdate() without an Exception after a FindRecord() call, allthough "out of range" at the "before first" position.

4. SystemParameters.AlternateDatabaseRecoveryPath
Martin, maybe ist is possible to add the AlternateDatabaseRecoveryPath to SystemParameters to allow to open a relocated (moved/copied) ESE database!?
        /// <summary>
        /// This parameter can be used to force crash recovery or a restore operation to look for the databases referenced in the transaction log in the specified folder. 
        /// </summary>
        public string AlternateDatabaseRecoveryPath
        {
            get
            {
                int ignored = 0;
                string val;
                Api.JetGetSystemParameter(this.instance.Inst, JET_SESID.Nil, Server2003Param.AlternateDatabaseRecoveryPath, ref ignored, out val, 1024);
                return val;
            }

            set
            {
                Api.JetSetSystemParameter(this.instance.Inst, JET_SESID.Nil, Server2003Param.AlternateDatabaseRecoveryPath, 0, value);
            }
        }
Kind Regards,
Tom

New Post: Test driving Microsoft.Database.Isam

$
0
0
Hi Tom. Lots of different topics here. It may get confusing. In the future, it may be easier if you open a new Discussion Topic for each topic. I don't want to drop anything unintentionally!
  1. I actually didn't write the first version of Isam. :) The person who did used DBNull instead of Nullable Types. I wonder if that would have been cleaner.
  2. Yes, that is correct!
  3. Are you saying that the problem is after every EditRecord after a FindRecord? Or only if it's at the end of the desired range? I'm trying to build up a suite of tests, and want to make sure your scenario is covered.
  4. I'll also have to add a test for that.
  5. We've done some restructuring with system parameters to try and keep Isam simple, and also share some common coding with PersistentDictionary. Both of them will use a new class called 'IsamSystemParameters'. Unfortunately, they still don't have AlternateDatabaseRecoveryPath...
    I'm debating whether this is a common enough scenario to justify adding yet another parameter to IsamSystemParameters. It's a balance between simplicity and control.
Thanks for the great feedback everyone!


-martin

New Post: About Esent projects and alike

$
0
0
So I invested considerable amount of time researching different storage engine options for a backend service I’m developing and here are my findings:
  1. $50 – 100K Server database products for which a fraction of their feature set is relevant to my needs and even for that I need to struggle for it to work as I want. No limitations on cpu / memory.
  2. $2000 – 3000 Server Database products for which a fraction of their feature set is relevant to my needs. Limitations imposed on cpu / memory.
  3. "Free" Server database products similar to (1) without limitations on the cpu/ memory but come with an inconclusive license agreement as far as commercial-use or high paid support cost, at least for startups, which ultimately are calculated based on hardware as (1), alternatively, community support means: publicly disclose your code (fair request for help) maybe you get an answer maybe you don't, maybe the client library will work fine in .net (if one exists) maybe not…
  4. Free "client databases" which seem great for typical client-side caching / low processing requirements, worst case scenario the user will clear the cache.. not a big deal.
  5. Various “Question mark products” – Embedded storage engines that appear to have proven reputation in various server projects, seem to serve 90% of my needs but appear to lack a firm standing commercial-wise – ESENT:
    • Is there a dedicated team behind this product or do Exchange have their own fork , IE have another and everyone is trying to make sense of what someone wrote 20 years ago..?
      Did someone replace the person who left “unexpectedly” to cont. working on the JetResizeDatabase api problem? http://managedesent.codeplex.com/discussions/433309
    • Has the managed wrapper been “upgraded” to MSDN and bundled with Win8 as a sign of future commitment and continued development or just out of convenience for MS internal projects? Is this just an experimental side CodePlex project (e.g if it works for you great). Why isn’t it on the MSDN front page? http://managedesent.codeplex.com/discussions/455631
    • What’s the expected support policy as far as .NET? Will it work with .NET 5? (I imagine being part of windows the native part is tied to the OS support policy, but is there an option to open a ticket on the .NET wrapper if an A/V happens http://managedesent.codeplex.com/discussions/522436
What drives me nuts is it appears there are very good free “storage building blocks” that are either risky or are support costly by who knows who… So I ask myself why FREE?! Why not REASONABLEY PRICED FOR CONTINUED MAINTAINE? Can you imagine a serious company (that needs such tech) that won’t pay $ 1k – 5k a year to assure there is financial justification to keep the people working on it happy/employed? Let me be clear, I’m NOT talking about: personal end-user support. training, design, custom development, and especially new features – just that it is well documented, works in a performant manner, blocking bugs get fixed and it will cont. to work in newer development environments as they come out.

“Dll as a Service“ :)

Hope the message bubbles up.

-Itai

Created Unassigned: Isam: Creating IsamInstance without trailing slash '\' for Working path causes unexpected errors down stream, constructor should validate assumption. [13391]

$
0
0
Repro:

IsamInstance i = new IsamInstance(@"C:\foo\bar", false);

Later...

var idx = new IndexDefinition(PrimaryIndex);
idx.Flags = IndexFlags.Primary | IndexFlags.Unique;
idx.KeyColumns.Add(new KeyColumn(IdentityColumn, true));
defininition.Indices.Add(idx);

database.CreateTable(defininition);

This will cause EsentFileAccessDeniedException to be thrown, which is kind of hard to wrap your head around.

I notice in the doc comments it calls for a trailing slash to be added to the working directory when constructing IsamInstance, it would be great if IsamInstance could validate the assumption, and if it fails - throw. It would have saved me a ~1 hour of digging around trying to figure out what was going on :-)

New Post: PersistentDictionary and Transactions

$
0
0
So no movement here? Is TransctionScope supported?

New Post: ESENT for web app

$
0
0
Has anyone tried ESENT as a store for relatively small web app? I'm concerned about its way of transactions isolation. There seems to be no way to lock row for read so that multiple threads didn't overwrite each other changes. Or maybe ESENT is not an option for Web?

New Post: ESENT for web app

$
0
0
JetGetLock() exists ( see https://msdn.microsoft.com/en-us/library/gg294094(v=exchg.10).aspx ), but are you sure it's what you want?

Usually people like the fact that different transactions don't block each other, as long as they don't write the same cell.
If two different transactions do try that, then the second gets a JET_errWriteConflict/EsentWriteConflictException. The second transaction can then rollback, and try again.

-martin

New Post: ESENT for web app

$
0
0
For some time, comments on my personal web site were stored in ESENT DB.

I disabled the comments because I did not want to fight spammers; however, it worked well for a few years on my shared asp.net hosting.

New Post: Problem querying a IE 11 WebCacheV01.dat file

$
0
0
Hi,
I am trying to query some tables on a IE 11 WebCacheV01.dat file but looks like ESENTManaged cannot find records on some of them. I have opened the same db file on NirSoft ESEDatabaseView and it can find records on them.

For both History tables my script cannot go to the first record using TryMoveFirst, so it thinks the table is empty.

Here is my code:
using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.Isam.Esent.Interop;

namespace ConsoleESEDBApplication
{
    class Program
    {
        static void Main(string[] args)
        {

            int pageSize;
            String fileName = @"C:\Users\dummy\Downloads\WebCacheV01.dat";

            try
            {
                Api.JetGetDatabaseFileInfo(fileName, out pageSize, JET_DbInfo.PageSize);
                SystemParameters.DatabasePageSize = pageSize;

                using (var instance = new Instance("SuperTimeline"))
                {
                    var param = new InstanceParameters(instance);
                    param.Recovery = false;
                    instance.Init();
                    using (var session = new Session(instance))
                    {
                        Api.JetAttachDatabase(session, fileName, AttachDatabaseGrbit.ReadOnly);
                        JET_DBID dbid;
                        Api.JetOpenDatabase(session, fileName, null, out dbid, OpenDatabaseGrbit.ReadOnly);

                        using (var tableContainers = new Table(session, dbid, "Containers", OpenTableGrbit.ReadOnly))
                        {
                            IDictionary<string, JET_COLUMNID> columnIds = Api.GetColumnDictionary(session, tableContainers);

                            if (Api.TryMoveFirst(session, tableContainers))
                            {
                                do
                                {

                                    var containerId = (int)Api.RetrieveColumnAsInt32(session, tableContainers, columnIds["ContainerId"]);
                                    var name = Api.RetrieveColumnAsString(session, tableContainers, columnIds["Name"], Encoding.Unicode);

                                    using (var table = new Table(session, dbid, "Container_" + containerId, OpenTableGrbit.ReadOnly))
                                    {
                                        if (Api.TryMoveFirst(session, table))
                                        {
                                            Console.WriteLine(name + " | Container_" + containerId + " | has records!!!");
                                        }
                                        else
                                        {
                                            Console.WriteLine(name + " | Container_" + containerId + " | does not have records!!!");
                                        }
                                    }

                                } while (Api.TryMoveNext(session, tableContainers));
                            }
                        }
                    }
                }

            }
            catch (Exception e)
            {
                Console.WriteLine(e.Message);
            }

            Console.ReadLine();

        }
    }
}

New Post: Problem querying a IE 11 WebCacheV01.dat file

$
0
0
I'm not sure whether it's good news or bad news, but your code works fine for me:
History\0 | Container_1 | has records!!!
Content\0 | Container_2 | has records!!!
iecompat\0 | Container_3 | does not have records!!!
No such table or object
And the 'Container_3' table actually has zero records, so that is correct.
It fails trying to open 'Container_4', and for my database, there is no 'Container_4'. So that's expected, too.


-martin

New Post: Problem querying a IE 11 WebCacheV01.dat file

New Post: Database corruption issue

$
0
0
I've received a corrupt database from a customer and my attempts at repairing the corruption has been unsuccessful. The corruption occurred on a Windows 2012 R2 machine, 32bit application. ESE killed itself due to "a transient memory corruption"†. The next JetInit after that failed. Before I received the database, they attempted to fix the database, so I don't know if/how things were further broken. One possible interesting thing is that the log signature for the database file is zeroed out.

Since I couldn't get esentutl /r to work to recover the database, I went onto attempts to /p it. The repair succeeded (with a warning that corruption existed, which was expected, I think). At this point, the primary / only index on the largest table didn't exist. An attempt to create that index resulted in an invalid parameter exception. Subsequent database attach attempts failed with invalid parameter, seemingly coming out from ErrDBTryCreateSystemTable (I didn't try to go deeper into that path). After going back to the original database and doing another repair, I attempted readonly access, and while I could open the table, TryMoveFirst / TryMoveNext failed with InvalidParameter. Subsequent repair attempts report nothing wrong, just this warning:
WARNING: Database file is too big (expected 33640176 pages, file is 33640432 pages)
The integrity check of that table takes a long time, and coupled with the "database file being too big" only being by 256 pages, I'm assuming that most of the data is still there and reachable, I'm just stumped for where to go from here. Attempting an offline defrag out of desperation results in an illegal duplicate key. I'm currently running an integrity check. Previous checksum validations passed fine.

Thank you,
Michael


† I'm not using ManagedESENT, so I'm assuming in addition to hardware issues, wild pointers are also a reason for the issue. I'm currently trying to find the cause of that problem. Is the detection new to Server 2012? Would the way that the process aborts create a crash dump if something was configured the right way? Either Windows-specific or an ESE setting?

Created Unassigned: Bug: TrySeekTruncatedString() fails when primary key contains "-" character and key is truncated with cbkeymost set to maximum [13461]

$
0
0
Hi,

I'm inserting following string to database and then use TrySeekTruncatedString() to find record but it returns false:
* string key = "-" + new string('a', 600);

The problem occurs when I set cbKeyMost to 1000 (bytes) and save a string larger (>500 chars) so that truncated portion of key includes "-" character.

Not sure, if I'm doing something wrong. Any help will be appreciated. Thanks.

Example:

var columndef = new JET_COLUMNDEF()
{
cp = JET_CP.Unicode,
coltyp = JET_coltyp.LongText,
grbit = ColumndefGrbit.ColumnNotNULL
};
Api.JetAddColumn(sesid, tableid, "KeyColumn", columndef, null, 0, out keyColumnid);


string indexDef = "+KeyColumn\0\0";
var indexcreate = new JET_INDEXCREATE
{
szIndexName = "primary",
szKey = indexDef,
cbKey = indexDef.Length,
grbit = CreateIndexGrbit.IndexPrimary,
ulDensity = 100,
cbKeyMost = 1000,
cbVarSegMac = 1000,
};

New Post: Problem querying a IE 11 WebCacheV01.dat file

$
0
0
You code is correct. Nirsoft is not.
esentutl -mm -v Administrator_WebCacheV01.dat
******************************* META-DATA DUMP *******************************
Name                                               Type    ObjidFDP    PgnoFDP
==============================================================================
Admin                                               Db            1          1
...
Container_4                                         Tbl          11         45
  HashEntryIdIndex                                  Pri          11         45
From that, we see that that the first page is page #45. So let's dump that:
esentutl -mm -v Administrator_WebCacheV01.dat -p45
TAG   0    cb:0x0010    ib:0x0000    offset:0x0050-0x0060    flags:0x0000 (   )
TAG   1    cb:0x0210    ib:0x02b9    offset:0x0309-0x0519    flags:0x0003 (vd )
TAG   2    cb:0x01ed    ib:0x04c9    offset:0x0519-0x0706    flags:0x0003 (vd )
TAG   3    cb:0x02a9    ib:0x0010    offset:0x0060-0x0309    flags:0x0003 (vd )
TAG   4    cb:0x02a9    ib:0x115b    offset:0x11ab-0x1454    flags:0x0003 (vd )
TAG   5    cb:0x02a1    ib:0x1404    offset:0x1454-0x16f5    flags:0x0003 (vd )
TAG   6    cb:0x02a1    ib:0x06b6    offset:0x0706-0x09a7    flags:0x0003 (vd )
TAG   7    cb:0x02a1    ib:0x0957    offset:0x09a7-0x0c48    flags:0x0003 (vd )
TAG   8    cb:0x02c2    ib:0x0bf8    offset:0x0c48-0x0f0a    flags:0x0003 (vd )
The 'd' in the flags means 'deleted'. ESE will 'soft-delete' the records first, and then later clean them up. These rows have been 'soft-deleted', and are therefore not valid.

Nirsoft says "esent.dll (The dll file of Extensible Storage Engine) is not required to read the database". That means they've written their own code to access the database file, and can get differing results.

-martin
Viewing all 496 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>