Elite: Berbahaya dan CosmosDB

o7 cmdr!

Pada malam karantina yang hangat, di salah satu obrolan telegram di Elite: Dangerous, sebuah diskusi berkobar tentang topik: bintang jenis apa yang paling sering memiliki planet mirip Bumi?

Faktanya adalah eksplorasi planet adalah salah satu mekanisme utama permainan. Dan dalam hierarki kegunaan planet, planet yang mirip Bumi berada di urutan paling atas. Namun kelangkaannya juga cukup tinggi. Jadi para komandan ingin tahu: bintang mana yang harus diperhatikan saat bergerak di sepanjang Bima Sakti?

Dari diskusi ini, lahirlah sebuah proyek, yang akhirnya saya kubur. Tidak, kami menemukan jawaban atas pertanyaan yang diajukan dengannya. Tetapi saya tidak menyukai proyek tersebut karena berbagai alasan, dan setelah beberapa bulan menunda-nunda, saya meluncurkan iterasi kedua. Apa yang muncul dari ini, serta jawaban atas pertanyaan yang diajukan - dalam artikel ini.

pengantar

Di elit, sebagian besar gameplay terjadi, apa pun yang Anda pikirkan, di browser dengan selusin tab terbuka. Kami tidak akan memperdebatkan apakah itu benar atau tidak, tetapi, yang pasti, ini memiliki daya tarik tersendiri dan banyak orang menyukainya.

, , . , Frontier Developments . , .

? , Log- . .

() ( Windows):

C:\Users\User Name\Saved Games\Frontier Developments\Elite Dangerous\

. . .. PC ( ) — .

, , , . ? - , ( , , , , POI, etc).

PC:

. - . , . , , . , 400 (, ! ). , -.

, . ( , ). : , , (, ) .. , , , , — EDSM EDDB.

, , — . .

EDSM , . , , . , , — JSON 10. - - . , , - , . , ! .

% . 6 . .

, , . , , !

, , , - ( , ). EDSM, EDDB, etc. — EDDN. , , , .

, EDDN ? ...

EDDN

EDDN — :

The Elite: Dangerous Data Network is a system for willing Commanders to share dynamic data about the galaxy with others.

By pooling data in a common format, tools and analyses can be produced that add an even greater depth and vibrancy to the in-game universe.



EDDN is not run by or affiliated with Frontier Developments.

( ).

HTTP Endpoint https://eddn.edcd.io:4430/upload/ ( ).

. ZeroMQ tcp://eddn.edcd.io:9500. .

( ):

, . ? , . , ( ) ! , - .

, :

— Microsoft Azure. - . — Azure, dotnet core/standard

--- , .

, :

1. Message Distributor

EDDN Channel . ( schemaRef , ) — . EDDN 5:

• https://eddn.edcd.io/schemas/journal/1
• https://eddn.edcd.io/schemas/blackmarket/1
• https://eddn.edcd.io/schemas/commodity/3
• https://eddn.edcd.io/schemas/outfitting/2
• https://eddn.edcd.io/schemas/shipyard/2

. ? , ( ), , . journal, .. ( ). , , .

, MessageDistributor EDDN . .

2. Azure Storage Queue

( Message Processor , ). , . Storage Account ( ) connection string ( , Azure AD, ). Storage Account ( — , , ).

Azure Storage Emulator Azure Storage Explorer

3. Message Processor

, , . Azure Function App.

, - ? .

, Azure Function WebJob ( , ), . , Azure Function Runtime ( , ) — — , — . , .. — .

, , (Scale Out), , , - , CPU/Memory consumption, etc.

, (in/out bindings). , , QueueTrigger — (, dotnet, ). CosmosDBTrigger . ( ). , , CosmosDB ( ) . - , , : db client ( in-built DI, ). , , queue .

QueueTrigger. , — (invisible) . , 30 ( ). 30 — DequeueCount 1. ( -> visible state on, ++DequeueCount). DequeueCount = 5 ( ), -poison. journal, 5 , journal-poison. ( - ). , . .

: WebJob AppService. — Function App. : , .

. : , , 2 . 2 , , . , (upsert — update || insert) , , ;-) 2 , , . , . , . , , , . , - (, , CosmosDB ACID compliant).

IMHO: , . . 1. , — . (, ) .

Cosmos DB Optimistic Concurrency, . . .

4. CosmosDB

. , ( ) , . , .. .

4: Signals, Systems, Stations, Bodies. , journal ( , ? -).

, journal Event, : CarrierJump, Docked, FsdJump, Location, SaaSignalsFound, Scan ( ). , — . : — . .

CosmosDB. ( , ) Request Unit per second — RU/s. .. RU/s . 2 : Provisioned throughput Serverless ( , ). , Provisioned RU/s ( ), Serverless, RU/s , , .



Provisioned RU/s. , RU/s 400, 10% 1000% (40-4000 RU/s)



RU/s Provisioned mode — 400. , , , 250 RU/s 150 , Serverless .



, ( CosmosDB). : CosmicClone. Serverless . 2 . Provisioned Mode , , .



, zero downtime "" — , MessageDistributor , . , . . .



CosmosDB

, , : EDDN , .

, React Native — dotnet, . .

: EDDNConsumer. , , .

:

• EDDNConsumerCore — MessageDistributor. EDDN
• EDDNModels — EDDN
• JournalContributor — MessageProcessor journal
• SharedLibrary — . .

EDDNConsumerCore

dotnet core 3.1 . Main DI. HostedService ConsumerService

services.AddHostedService<ConsumerService>();

nuget- ( ): NetMQ — ZeroMq Ionic.Zlib — NetMQ

StartAsync, — NetMQRuntime ClientAsync ( , ):

private async Task ClientAsync()
{
    var utf8 = new UTF8Encoding();
    using (var client = new SubscriberSocket())
    {
        client.Connect(_eddnClientSettings.ConnectionString);
        client.SubscribeToAnyTopic();
        while (true)
        {
            try
            {
                (var bytes, _) = await client.ReceiveFrameBytesAsync();
                var uncompressed = ZlibStream.UncompressBuffer(bytes);
                var result = utf8.GetString(uncompressed);
                await _messageDistributor.DistributeAsync(result);
                _logger.LogInformation(result);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error reading message queue");
            }
        }
    }
}

, ( , 1 ), _messageDistributor , , . .

MessageDistributor DistributeAsync:

public async Task DistributeAsync(string message)
{
    try
    {
        using var stringReader = new StringReader(message);
        using var jsonReader = new JsonTextReader(stringReader);
        var result = _serializer.Deserialize<Entity<BaseMessage>>(jsonReader);
        var queue = await _messageQueueFactory.GetQueueAsync(result);
        await queue.SendMessageAsync(message.Base64Encode());
    }
    catch(Exception ex)
    {
        _logger.LogError(ex, "Error distributing message");
    }
}

, , MessageQueueFactory. — . : . , schemaRef header ( test_data.json ). , , .

, ( — JSON) , . \\ , — , , null default. ( ) , POCO , , , , - . .

, , json- NewtonsoftJson ( json, ). Missing Member

_serializer.MissingMemberHandling = MissingMemberHandling.Error;
_serializer.Error += _serializer_Error;

, Application Insights, Notification Alerts . , , Header. . , .. , missing property. , , .

, header , . , ( ):

public class MessageQueueFactory : IMessageQueueFactory
{
    private readonly StorageAccount _storageOptions;
    private readonly QueueMapping _queueMapping;
    private readonly IDictionary<string, QueueClient> _queues = new Dictionary<string, QueueClient>();

    public MessageQueueFactory(
        IOptions<StorageAccount> storageOptions,
        IOptions<QueueMapping> queueMapping)
    {
        _storageOptions = storageOptions.Value;
        _queueMapping = queueMapping.Value;
    }

    public async Task<QueueClient> GetQueueAsync(Entity<BaseMessage> entity)
    {
        if (_queueMapping.TryGetValue(entity.SchemaRef, out var queueName))
        {
            if (!_queues.ContainsKey(queueName))
            {
                var client = new QueueClient(_storageOptions.StorageConnectionString, queueName);
                await client.CreateIfNotExistsAsync();
                _queues.Add(queueName, client);
            }
            return _queues[queueName];
        }
        throw new ArgumentException($"Queue {entity.SchemaRef} has not configured", nameof(entity.SchemaRef));
    }
}

, appsettings json :

{
  "QueueMapping": {
    "eddn.edcd.io/schemas/journal/1": "journal"
  }
}

( ):

"QueueMapping": {
    "eddn.edcd.io/schemas/journal/1": "journal",
    "eddn.edcd.io/schemas/blackmarket/1": "blackmarket",
    "eddn.edcd.io/schemas/commodity/3": "commodity",
    "eddn.edcd.io/schemas/outfitting/2": "outfitting",
    "eddn.edcd.io/schemas/shipyard/2": "shipyard"
  }

Dictionary<string, string> schemaRef. .

1 _queues — QueueClient, . , . , . round trip , . , , . , .

, — , — . . MessageDistributor, .

. 3 . .

EDDNModels

POCO . , . , JournalMessage:

[JsonProperty("id")]
public override string Id
{
    get => Event switch
    {
        JournalEvent.FsdJump => StarSystem,
        JournalEvent.Scan => BodyName,
        JournalEvent.Docked => StationName,
        JournalEvent.Location => BodyName,
        JournalEvent.CarrierJump => StarSystem,
        JournalEvent.SaaSignalsFound => BodyName,
        _ => $"UnknownID_{Guid.NewGuid()}"
    };
}

Id, , CosmosDB. ORM. , journal ? , . Event Id. - , - , - .

. : , , , , , , , , , \ .. ( JournalMessage). , , .. POCO — .

.

JournalContributor

"" journal — Azure Function App. Function App dotnet core class library , , . FunctionName . : — Function App ( ). - — . , . . , , ( ).

, :

public class JournalContributor
    {
        private readonly IEventTypeProcessorFactory _eventTypeProcessorFactory;

        public JournalContributor(IEventTypeProcessorFactory eventTypeProcessorFactory)
        {
            _eventTypeProcessorFactory = eventTypeProcessorFactory;
        }

        [FunctionName("ContributeJournal")]
        public async Task Run(
            [QueueTrigger("journal", Connection = "AzureWebJobsStorage")]
            Entity<JournalMessage> myQueueItem,
            ILogger log)
        {
            try
            {
                var eventProcessor = _eventTypeProcessorFactory.GetProcessor(myQueueItem.Message);
                await eventProcessor.ProcessEventAsync(myQueueItem.Message);
            }
            catch(Exception ex)
            {
                log.LogError(ex, $"Error processing queue item: {JsonConvert.SerializeObject(myQueueItem)}");
                throw;
            }
        }
    }

QueueTrigger ( ). , Event … .

, , . — , dotnet . Startup Configure, FunctionStartup (assembly):

using JournalContributor.Settings;
using Microsoft.Extensions.Configuration;
using System.IO;

[assembly: FunctionsStartup(typeof(JournalContributor.Startup))]

namespace JournalContributor
{
    public class Startup : FunctionsStartup
    {
        private IConfigurationRoot _functionConfig;
        private readonly string COSMOS_CONNECTION_STRING = Environment.GetEnvironmentVariable("CosmosDBConnectionString");
        private readonly string ENVIRONMENT = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");

        public override void Configure(IFunctionsHostBuilder builder)
        {
            _functionConfig = new ConfigurationBuilder()
                .AddJsonFile(Path.Combine(builder.GetContext().ApplicationRootPath, "appsettings.json"), optional: true, reloadOnChange: true)
                .AddJsonFile(Path.Combine(builder.GetContext().ApplicationRootPath, $"appsettings.{ENVIRONMENT}.json"), optional: true, reloadOnChange: true)
                .Build();

            builder.Services.AddSingleton<CosmosClient>(factory => new CosmosClient(COSMOS_CONNECTION_STRING));
            builder.Services.AddSingleton<IEventTypeProcessorFactory, EventTypeProcessorFactory>();
            builder.Services.AddTransient<FsdJumpProcessor>();
            builder.Services.AddTransient<ScanProcessor>();
            builder.Services.AddTransient<DockedProcessor>();
            builder.Services.AddTransient<LocationProcessor>();
            builder.Services.AddTransient<SaaSignalsFoundProcessor>();
            builder.Services.Configure<CosmosDbSettings>(_functionConfig.GetSection("CosmosDbSettings"));
        }
    }
}

DI . , CosmosClient : (CosmosDB). ? . 2 CosmosDB — , . , .

. , , , switch :

public IEventTypeProcessor GetProcessor(JournalMessage journalMessage) => journalMessage.Event switch
        {
            JournalEvent.FsdJump => _serviceProvider.GetService<FsdJumpProcessor>(),
            JournalEvent.Scan => _serviceProvider.GetService<ScanProcessor>(),
            JournalEvent.Docked => _serviceProvider.GetService<DockedProcessor>(),
            JournalEvent.Location => _serviceProvider.GetService<LocationProcessor>(),
            JournalEvent.SaaSignalsFound => _serviceProvider.GetService<SaaSignalsFoundProcessor>(),
            JournalEvent.CarrierJump => _serviceProvider.GetService<DockedProcessor>(),
            _ => throw new ArgumentException($"Unknown Event {journalMessage.Event}", nameof(journalMessage.Event))
        };

, , . ScanProcessor, .. 2- :

public async Task ProcessEventAsync(JournalMessage message)
        {
            var existingItem = await message.CheckIfItemExists(_bodiesContainer, _options.BodiesCollection.PartitionKey);
            if (existingItem == null)
                await _bodiesContainer.UpsertItemAsync(message);
            else
            {
                //Basic scan type contains less information then others.
                //We`re skipping item upsert if remote item has scan type higher then Basic
                //We`re also skippings if remote item scan type is Detailed (as it`s a maximum info scan)
                //Will update item if both (remote and current) have scan type Basic
                //And will upsert item in case of current item have higher scan type then remote
                if ((message.ScanType == ScanType.Basic && existingItem.ScanType != ScanType.Basic) ||
                    existingItem.ScanType == ScanType.Detailed)
                    return;
                else
                    await _bodiesContainer.UpsertItemAsync(message);
            }
        }

, , ( !) , , , , . . . , . , : , Bodies Id . , .

UpSert, Insert, .. concurrency : Item with such Id already exists ( - ). , , Optimistic Concurrency . . upsert.

, ScanType. Detailed — . Basic — — . Basic — . .

. , 3 , MessageProcessor journal .

, . , Azure WebJob EDDN , Storage Account , Azure Function App Cosmos DB . Application Insights (, -, , ). , .

continuous, .. .

![image](img src="https://habrastorage.org/webt/i_/5u/zq/i_5uzqxkanblmotxacqhtwnp_wq.png)

, , schemaRef . , .

Stream-log : , , … .

. , , .

RU/s . — Provisioned Mode 400 RU/s. , . ~105RU/s.

. 145 . EDSM, .

. Check.

. . .

. . - .

45 . , MSDN , 45 . . , . . 45 — 1.5 . , — .

— . - — , - — "" . .

. ~10 . appinsjournalweprivate . Function App. — , . \. :

( ), host.json JournalContributor:

{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingExcludedTypes": "Request",
      "samplingSettings": {
        "isEnabled": true
      }
    },
    "logLevel": {
      "default": "Information",
      "Host": "Error",
      "Function": "Error",
      "Host.Aggregator": "Information"
    }
  }
}

:

, ( 21 ):

( 193 ). - / . 21 1.75 . — appinsweprivate — WebJob. 0.98 0.19 21 .

cdbweprivate — CosmosDB Provisioned Mode 400 RU/s. . cdbslweprivate — Serverless mode CosmosDB. 4.47 .

— laweprivate. Log Analytics Workspace, , , .. CosmosDB. , , , . .

, 175,8 . , .

, . -. appinsights — 3 . , . , % , . - , , 3 — ( ).

-. CosmosDB RU/s ( , ) RU/s (Pricing), :

EDDN. , , Elite: Dangerous. - " " — . , , DLC Odyssey , . .

, Function App Storage Account, , .

, . — .

. .

.

" " " ", . - , . =) , -, ...

— . , - , 145 , EDSM 57 . - - , .

, , , . . " " — .

, — . ? . , , EDSM . , JSON — . — ( ). RU/s , , . .

CosmosDB — Jupyter Notebooks. C#, Python, - . , .

, , . ( , Sol [0,0,0]:

? :

5 :

EDSM, - , . , , — - . , , .

CosmosDB, Azure Function Change Feed ( ), .

— CosmosDB .

Bagi mereka yang telah bertahan sampai akhir dan masih tertarik untuk mencari planet mirip bumi di kalangan elit, berikut adalah jawaban Anda (hasil berdasarkan data yang diperoleh dari dump EDSM lengkap sekitar enam bulan yang lalu):

Jenis F, K, G, A dan, untuk beberapa alasan, neutron membentuk 5 jenis bintang teratas tempat pilot menemukan planet mirip Bumi. Nah, inilah hasilnya ¯ \ _ (ツ) / ¯

Baik. Itu saja.

Terbang dengan aman, cmdr!